4.1.3
Allowed setting the target table to Auto Created Table when a StarRocks database was the target end.
4.1.7.2
Supported the Mark Timestamp During Synchronization function in pipeline tasks for the StarRocks database of V2.1 and later versions.
4.1.10.1
Supported the Synchronize Source Table Structure Change function when a StarRocks database was the target end of a pipeline task.
Scheduled Task supported automatic synchronization of the primary key setting of the source table to the target table when a StarRocks database was the target end and the target table was set to Auto Created Table.
In scheduled tasks and pipeline tasks, the source data containing separators could be written successfully to a StarRocks database of V3.0 or later versions.
4.1.11.5
When you set Configuration Method in Data Source to Table Selection, and set Target Table and Table Type in Data Destination and Mapping to Auto Created Table (in a StarRocks database) and Duplicate Key, respectively, the Not Null information of the source table fields would be displayed in the Field Mapping area.
4.2.1.4
The DB Table Output operator in real-time tasks in Data Development supported data insertion without a primary key.
4.2.3.1
When a StarRocks database was the target end of a scheduled task in Data Development:
You could set Column Separator and Line Separator.
You could not set decimal-type fields as the primary key of the target table in Data Destination and Mapping when Table Type of the target table was set to Primary Key and the source table had no primary key.
FineDataLink supports connection to StarRocks databases, which can be used as the source and target ends in scheduled tasks and as the target end in pipeline tasks.
Data Source
When reading data from a StarRocks database, ensure that empty strings are enclosed in single quotes, not double quotes.
Target Table Set to Auto Created Table
If StarRocks is selected as the data destination, you need to select a table type when configuring the target table, as shown in the following table.
StartRocks supports auto-created tables, where the table names must begin with an English letter. Otherwise, the table creation will fail.
In V4.1.10.1 and later versions, when StarRocks serves as the target end and the target table is set to Auto Created Table, the primary key in the source end will be automatically marked as the primary key of the auto-created table.
Duplicate Key
You can tick/untick NOT NULL.
Unique Key
You must set the physical primary key.
Primary Key
This table type is supported for StartRocks of V1.19 and later versions.
In FineDataLink V4.2.3.1 and later versions, for a source table with no primary key, you cannot set decimal-type fields as the primary key of the target table in Data Destination and Mapping.
Aggregate Key
This table type is not supported for auto-created tables.
Target Table Set to Existing Table
If StarRocks is set as the data destination, you need to select a table type when configuring the target table. Certain restrictions apply when you configure the write method, as shown in the following table.
2. You cannot select Add/Modify/Delete Data Based on Identifier Field as Write Method.
1. When you select Write Data into Target Table Directly as Write Method, you can select only Overwrite Data in Target Table If Same Primary Key Value Exists as Strategy for Primary Key Conflict.
The following strategies are not supported:
Ignore Source Data If Same Primary Key Value Exists
Record as Dirty Data If Same Primary Key Value Exists
2. For Unique Key, if Write Method is set to Add/Modify/Delete Data Based on Identifier Field, only physical deletion is supported. (The logical deletion is not supported.)
1. When you select Write Data into Target Table Directly as Write Method, you cannot configure Primary Key Mapping and Strategy for Primary Key Conflict.
Separator Description
Versions earlier than V4.1.10.1
During data writing to StarRocks, FineDataLink uses the default separator to split row and column data. If the data to be written contains the default separator, an error will occur during data writing.
Example
Data to be written:
2
3
If FineDataLink uses a comma (,) as the separator, 1,1 in column A will be split into two parts (1 and 1) during data writing, causing a write error.
Suggestion
You can first check whether the binary fields are synchronized from the source. If they are, you should disable synchronization for the binary fields, and use a formula to replace the separators (for example, replacing the separators with spaces).
From V4.1.10.1 to V4.2.3.1
Versions earlier than V3.0
V3.0 and later versions
V4.2.3.1 and later versions
You can specify the line and column separators, as shown in the following figure.
Line and column separators cannot be the same.
Column Separator Description
Supported options in the drop-down list: Automatic Processing, Recommended Separator, Comma, Tab, Semicolon, Vertical Bar, and ASCII Character
Advantage:
If the source data contains the default separators, FineDataLink will automatically escape and replace them (using escape characters to escape the line and column separators in the data), ensuring successful data writing.
Disadvantage:
The write performance may be compromised.
Recommended Separator (\x01)
In most scenarios, user data do not contain \x01. Therefore, this separator can avoid write errors while ensuring performance.
If the user data contains \x01, FineDataLink will not escape or replace \x01, and a write error will occur.
Comma, Tab, Semicolon, Vertical Bar, and ASCII Character
If an error occurs when you use Recommended Separator, you can specify a separator based on the actual data.
Line Separator Description
Supported options in the drop-down list: CR+LF, LF, and ASCII Character
If StarRocks is set as the data destination, you need to select a table type during the target table configuration, as shown in the following table.
Logical primary keys are not supported when StarRocks serves as the output end of the pipeline task.
Displayed only and unselectable
You can set the physical primary key.
DDL Synchronization Description
In FineDataLink of versions earlier than V4.1.10.1, synchronizing DDL changes is not supported when StarRocks serves as the target end. In FineDataLink of V4.1.10.1 and later versions, synchronizing DDL changes is supported when StarRocks serves as the target end.
When StarRocks serves as the target end, the following DDL changes can be synchronized.
The data type of a field is converted from TINYINT/SMALLINT/INT/BIGINT to TINYINT/SMALLINT/INT/BIGINT/DOUBLE.
The data type of a field is converted from TINYINT/SMALLINT/INT/BIGINT/LARGEINT/FLOAT/DOUBLE/DECIMAL to VARCHAR.
The maximum length of VARCHAR-type data is modified.
The data type of a field is converted from VARCHAR to TINYINT/SMALLINT/INT/BIGINT/LARGEINT/FLOAT/DOUBLE.
The data type of a field is converted from VARCHAR to DATE. (Currently supported formats include %Y-%m-%d", "%y-%m-%d", "%Y%m%d", "%y%m%d", "%Y/%m/%d, "%y/%m/%d.) The data type of a field is converted from DATETIME to DATE (retaining only the year-month-day information, for example, 2019-12-09 21:47:05 to 2019-12-09). The data type of a field is converted from DATE to DATETIME (with hours, minutes, and seconds set to zero automatically, for example, 2019-12-09 to 2019-12-09 00:00:00).
The data type of a field is converted from FLOAT to DOUBLE.
The data type of a field is converted from INT to DATE. (If the INT-type data is invalid, the conversion will fail, and the data remains unchanged.)
Other DDL changes will trigger a task execution error, while DDL synchronization failures will be logged.
Description of Marking Timestamp During Synchronization
In FineDataLink of versions earlier than V4.1.7.2, the Mark Timestamp During Synchronization function is not supported for StarRocks in pipeline tasks. In FineDataLink of V4.1.7.2 and later versions, the Mark Timestamp During Synchronization function is supported for StarRocks of V2.1 and later versions.
Logical Deletion Description
When you set Data Source to StarRocks and select Logical Deletion at Target End (which will add a fdlmarked_deleted field to the target table) in a pipeline task, the initial startup of the pipeline task will be slow (potentially taking tens of seconds at most). This is because StarRocks locks the table for an extended period when updating the schema, impeding data writing to the target table.
V4.1.10.1 and later versions
Other Description
When you write data to a target table in StarRocks, both new and updated data are processed using INSERT operations via the Stream Load method.
If StarRocks is set as the data destination in the DB Table Output operator:
Existing Table
If Table Type is set to Duplicate Key:
Synchronization without a primary key is supported, where data from the INSERT event is inserted into the target table directly. The Primary Key column is not displayed in Field Mapping. Duplicate data may occur upon task restart. DELETE or UPDATE events will result in errors.
You cannot configure Primary Key Mapping and perform logical deletion in Write Method.
Auto Created Table
In FineDataLink of versions earlier than V4.2.1.4, Table Type can only be set to Primary Key. In FineDataLink of V4.2.1.4 and later versions, Table Type can be set to Duplicate Key.
You can set Table Type to Duplicate Key in Table Field Mapping. If Table Type is set to Duplicate Key:
For details, see StarRocks Data Connection.
For details, see Overview of Scheduled Task.
For details, see Overview of Data Pipeline.
For details, see Overview of Data Service.
滑鼠選中內容，快速回饋問題
滑鼠選中存在疑惑的內容，即可快速回饋問題，我們將會跟進處理。
不再提示
10s後關閉
Submitted successfully
Network busy