1. FineDataLink supports connection to the Doris database for data reading and writing using scheduled tasks, data writing using pipeline tasks and real-time tasks, and for releasing data services.
2. Ensure that you have ADMIN privileges on the Doris database for using FineDataLink functions normally.
3. During data writing to a Doris database, the target table name cannot contain Chinese characters.
When reading data from a Doris database, ensure that empty strings are enclosed in single quotes, not double quotes.
If Doris is selected as the data destination, you need to select a table type when configuring the target table, as shown in the following table.
You cannot set the physical primary key.
You can tick/untick NOT NULL.
In FineDataLink of V4.1.11.5 and later versions, when Configuration Method in Data Source is set to Table Selection, you can set Target Table to Auto Created Table in Data Destination and Mapping and synchronize the non-null information of the source table through field mapping.
You must set the physical primary key.
This table type is supported in Doris of V1.2.0 and later versions.
If Doris is selected as the data destination, you need to select a table type when configuring the target table. Certain restrictions apply when you configure the write method, as shown in the following table.
1. When you select Write Data into Target Table Directly as Write Method, you cannot configure Primary Key Mapping and Strategy for Primary Key Conflict.
2. You cannot select Add/Modify/Delete Data Based on Identifier Field as Write Method.
When you select Write Data into Target Table Directly as Write Method, you can only select Overwrite Data in Target Table If Same Primary Key Value Exists as Strategy for Primary Key Conflict.
The following strategies are not supported:
Ignore Source Data If Same Primary Key Value Exists
Record as Dirty Data If Same Primary Key Value Exists
配置主键映射以及主键冲突策略
Versions earlier than V4.1.10.1
/
During data writing to Doris, FineDataLink uses the default separator to split row and column data. If the data to be written contains the default separator, an error will occur during data writing.
Example
Data to be written:
1,1
2
3
If FineDataLink uses a comma (,) as the separator, 1,1 in Column A will be split into two parts (1 and 1) during data writing, causing a write error.
Suggestion
You can first check whether the binary fields are synchronized from the source. If they are, you should disable synchronization for the binary fields, and use a formula to replace the separators (for example, replacing the separators with spaces).
Versions from V4.1.10.1 to V4.2.3.1
Versions earlier than V2.0
V2.0 and later versions
If the source data contains the default separators, FineDataLink will automatically escape and replace them (using escape characters to escape the line and column separators in the data), ensuring successful data writing.
V4.2.3.1 and later versions
You could set Column Separator and Line Separator, as shown in the following figure.
Column Separator Description
Supported options in the drop-down list include Automatic Processing, Recommended Separator, Comma, Tab, Semicolon, Vertical Bar, and ASCII Character.
Automatic Processing
Advantage:
Disadvantage:
The write performance may be compromised.
Recommended Separator (\x01)
In most scenarios, user data do not contain \x01. Therefore, this separator can avoid write errors while ensuring performance.
If the user data contains \x01, FineDataLink will not escape or replace \x01, and a write error will occur.
Comma, Tab, Semicolon, Vertical Bar, and ASCII Character
If an error occurs when you use Recommended Separator, you can specify a separator based on the actual data.
Supported options in the drop-down list include CR+LF, LF, and ASCII Character.
Compared with Doris of V2.0 and later versions, the Automatic Processing option is not available in Column Separator in versions earlier than V2.0.
Assume that you select Write Data into Target Table Directly and Overwrite Data in Target Table If Same Primary Key Value Exists as Write Method and Strategy for Primary Key Conflict, respectively, and operate as follows when writing data to a Doris database:
1. Synchronize all fields of the source table to the target table. For example, synchronize the field A from the source table to the target table. After the task runs, the field A of the target table will have data.
2. In Field Mapping, cancel the mapping relationship of the field A, which means that the field A will not be synchronized.
After the task runs, the values of the field A in the target table are all NULL.. This issue will be resolved later.
Duplicate Key
Displayed only and unselectable
Unique Key - Merge on Read
Unique Key - Merge on Write
Aggregate Key
In FineDataLink of versions earlier than V4.1.10.1, synchronizing DDL changes is not supported when Doris serves as the target end. In FineDataLink of V4.1.10.1 and later versions, synchronizing DDL changes is supported when Doris serves as the target end.
When Doris serves as the target end, the following DDL changes can be synchronized.
The data type of a field is converted from TINYINT/SMALLINT/INT/BIGINT/LARGEINT/FLOAT/DOUBLE/DECIMAL to a numeric type with a larger range.
The data type of a field is converted from TINYINT/SMALLINT/INT/BIGINT/LARGEINT/FLOAT/DOUBLE/DECIMAL to VARCHAR.
The maximum length of VARCHAR-type data is modified.
The data type of a field is converted from VARCHAR to TINYINT/SMALLINT/INT/BIGINT/LARGEINT/FLOAT/DOUBLE.
The data type of a field is converted from VARCHAR to DATE. (Currently supported formats include "%Y-%m-%d", "%y-%m-%d", "%Y%m%d", "%y%m%d", "%Y/%m/%d", and "%y/%m/%d".)
The data type of a field is converted from DATETIME to DATE (retaining only the year-month-day information, for example, 2019-12-09 21:47:05 to 2019-12-09).
The data type of a field is converted from DATE to DATETIME (with hours, minutes, and seconds set to zero automatically, for example, 2019-12-09 to 2019-12-09 00:00:00).
The data type of a field is converted from FLOAT to DOUBLE.
The data type of a field is converted from INT to DATE. (If the INT-type data is invalid, the conversion will fail, and the data will remain unchanged.)
Except for DATE and DATETIME, all data types of a field can be converted to STRING, but the data type of a field cannot be converted from STRING to any other type.
Other DDL changes will trigger a task execution error, while DDL synchronization failures will be logged.
In FineDataLink of versions earlier than V4.1.7.2, the Mark Timestamp During Synchronization function is not supported for Doris in pipeline tasks. In FineDataLink of V4.1.7.2 and later versions, the Mark Timestamp During Synchronization function is supported for Doris of versions later than V2.0.
Logical primary keys are not supported.
If FineDataLink uses a comma (,) as the separator, 1,1 in column A will be split into two parts (1 and 1) during data writing, causing a write error.
V4.1.10.1 and later versions
The following table describes specific settings when you select Doris as the data destination in the DB Table Output operator.
Existing Table
No restrictions apply to Table Type.
If Table Type is set to Duplicate Key:
Synchronization without a primary key is supported, where data from the INSERT event is inserted into the target table directly. The Primary Key column is not displayed in Field Mapping. Duplicate data may occur upon a task restart. DELETE or UPDATE events will result in errors.
You cannot configure Primary Key Mapping and perform logical deletion in Write Method.
Auto Created Table
In FineDataLink of versions earlier than V4.2.1.4, Table Type can only be set to Primary Key. In FineDataLink of V4.2.1.4 and later versions, Table Type can be set to Duplicate Key.
You can set Table Type to Duplicate Key in Table Field Mapping. If Table Type is set to Duplicate Key:
滑鼠選中內容，快速回饋問題
滑鼠選中存在疑惑的內容，即可快速回饋問題，我們將會跟進處理。
不再提示
10s後關閉
Submitted successfully
Network busy