Note:Usage Restriction
1. FineDataLink supports connection to the Doris database for data reading and writing using scheduled tasks, data writing using pipeline tasks and real-time tasks, and for releasing data services.
2. Ensure that you have ADMIN privileges on the Doris database for using FineDataLink functions normally.
3. During data writing to a Doris database, the target table name cannot contain Chinese characters.
Data Development - Scheduled Task
Data Reading
When reading data from a Doris database, ensure that empty strings are enclosed in single quotes, not double quotes.
Data Writing
Target Table Set to Auto Created Table
If Doris is selected as the data destination, you need to select a table type when configuring the target table, as shown in the following table.
Note:
| Table Type | Field Mapping Configuration |
|---|---|
| Duplicate Key | You cannot set the physical primary key. You can tick/untick NOT NULL. In FineDataLink of V4.1.11.5 and later versions, when Configuration Method in Data Source is set to Table Selection, you can set Target Table to Auto Created Table in Data Destination and Mapping and synchronize the non-null information of the source table through field mapping. |
| Unique Key - Merge on Read | You must set the physical primary key. You can tick/untick NOT NULL. |
| Unique Key - Merge on Write | You must set the physical primary key. You can tick/untick NOT NULL. This table type is supported in Doris of V1.2.0 and later versions. |
| Aggregate Key | This table type is not supported for auto-created tables. |
Target Table Set to Existing Table
If Doris is selected as the data destination, you need to select a table type when configuring the target table. Certain restrictions apply when you configure the write method, as shown in the following table.
| Table Type | Write Restriction |
|---|---|
| Duplicate Key | 1. When you select Write Data into Target Table Directly as Write Method, you cannot configure Primary Key Mapping and Strategy for Primary Key Conflict. 2. You cannot select Add/Modify/Delete Data Based on Identifier Field as Write Method. |
| Unique Key - Merge on Write | When you select Write Data into Target Table Directly as Write Method, you can only select Overwrite Data in Target Table If Same Primary Key Value Exists as Strategy for Primary Key Conflict. The following strategies are not supported:
|
| Unique Key - Merge on Read | |
| Aggregate Key | 1. When you select Write Data into Target Table Directly as Write Method, you cannot configure Primary Key Mapping and Strategy for Primary Key Conflict. 配置主键映射以及主键冲突策略 2. You cannot select Add/Modify/Delete Data Based on Identifier Field as Write Method. |
Separator Description
| FineDataLink Version | Doris Version | Description | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
Versions earlier than V4.1.10.1 | / | During data writing to Doris, FineDataLink uses the default separator to split row and column data. If the data to be written contains the default separator, an error will occur during data writing. Example Data to be written:
If FineDataLink uses a comma (,) as the separator, 1,1 in Column A will be split into two parts (1 and 1) during data writing, causing a write error. Suggestion You can first check whether the binary fields are synchronized from the source. If they are, you should disable synchronization for the binary fields, and use a formula to replace the separators (for example, replacing the separators with spaces). | ||||||||
Versions from V4.1.10.1 to V4.2.3.1 | Versions earlier than V2.0 | |||||||||
V2.0 and later versions | If the source data contains the default separators, FineDataLink will automatically escape and replace them (using escape characters to escape the line and column separators in the data), ensuring successful data writing. | |||||||||
V4.2.3.1 and later versions | V2.0 and later versions | You could set Column Separator and Line Separator, as shown in the following figure. Note:Line and column separators cannot be the same.
Column Separator Description Supported options in the drop-down list include Automatic Processing, Recommended Separator, Comma, Tab, Semicolon, Vertical Bar, and ASCII Character.
Supported options in the drop-down list include CR+LF, LF, and ASCII Character.
| ||||||||
Versions earlier than V2.0 | Compared with Doris of V2.0 and later versions, the Automatic Processing option is not available in Column Separator in versions earlier than V2.0. |
Special Scenario Description
Assume that you select Write Data into Target Table Directly and Overwrite Data in Target Table If Same Primary Key Value Exists as Write Method and Strategy for Primary Key Conflict, respectively, and operate as follows when writing data to a Doris database:
1. Synchronize all fields of the source table to the target table. For example, synchronize the field A from the source table to the target table. After the task runs, the field A of the target table will have data.
2. In Field Mapping, cancel the mapping relationship of the field A, which means that the field A will not be synchronized.
After the task runs, the values of the field A in the target table are all NULL.. This issue will be resolved later.
Data Pipeline
Target Table Set to Auto Created Table
If Doris is selected as the data destination, you need to select a table type when configuring the target table, as shown in the following table.
| Table Type | Field Mapping Configuration |
|---|---|
Duplicate Key | Displayed only and unselectable |
Unique Key - Merge on Read | You must set the physical primary key. You can tick/untick NOT NULL. |
Unique Key - Merge on Write | You must set the physical primary key. You can tick/untick NOT NULL. This table type is supported in Doris of V1.2.0 and later versions. |
Aggregate Key | Displayed only and unselectable |
DDL Synchronization Description
In FineDataLink of versions earlier than V4.1.10.1, synchronizing DDL changes is not supported when Doris serves as the target end. In FineDataLink of V4.1.10.1 and later versions, synchronizing DDL changes is supported when Doris serves as the target end.
When Doris serves as the target end, the following DDL changes can be synchronized.
The data type of a field is converted from TINYINT/SMALLINT/INT/BIGINT/LARGEINT/FLOAT/DOUBLE/DECIMAL to a numeric type with a larger range.
The data type of a field is converted from TINYINT/SMALLINT/INT/BIGINT/LARGEINT/FLOAT/DOUBLE/DECIMAL to VARCHAR.
The maximum length of VARCHAR-type data is modified.
The data type of a field is converted from VARCHAR to TINYINT/SMALLINT/INT/BIGINT/LARGEINT/FLOAT/DOUBLE.
The data type of a field is converted from VARCHAR to DATE. (Currently supported formats include "%Y-%m-%d", "%y-%m-%d", "%Y%m%d", "%y%m%d", "%Y/%m/%d", and "%y/%m/%d".)
The data type of a field is converted from DATETIME to DATE (retaining only the year-month-day information, for example, 2019-12-09 21:47:05 to 2019-12-09).
The data type of a field is converted from DATE to DATETIME (with hours, minutes, and seconds set to zero automatically, for example, 2019-12-09 to 2019-12-09 00:00:00).
The data type of a field is converted from FLOAT to DOUBLE.
The data type of a field is converted from INT to DATE. (If the INT-type data is invalid, the conversion will fail, and the data will remain unchanged.)
Except for DATE and DATETIME, all data types of a field can be converted to STRING, but the data type of a field cannot be converted from STRING to any other type.
Other DDL changes will trigger a task execution error, while DDL synchronization failures will be logged.
Description of Marking Timestamp During Synchronization
In FineDataLink of versions earlier than V4.1.7.2, the Mark Timestamp During Synchronization function is not supported for Doris in pipeline tasks. In FineDataLink of V4.1.7.2 and later versions, the Mark Timestamp During Synchronization function is supported for Doris of versions later than V2.0.
Primary Key Description
Logical primary keys are not supported.
Separator Description
| FineDataLink Version | Doris Version | Description | ||||||
|---|---|---|---|---|---|---|---|---|
Versions earlier than V4.1.10.1 | / | During data writing to Doris, FineDataLink uses the default separator to split row and column data. If the data to be written contains the default separator, an error will occur during data writing. Example Data to be written:
If FineDataLink uses a comma (,) as the separator, 1,1 in column A will be split into two parts (1 and 1) during data writing, causing a write error. Suggestion You can first check whether the binary fields are synchronized from the source. If they are, you should disable synchronization for the binary fields, and use a formula to replace the separators (for example, replacing the separators with spaces). | ||||||
V4.1.10.1 and later versions | Versions earlier than V2.0 | |||||||
V2.0 and later versions | If the source data contains the default separators, FineDataLink will automatically escape and replace them (using escape characters to escape the line and column separators in the data), ensuring successful data writing. |
Data Development - Real-Time Task
The following table describes specific settings when you select Doris as the data destination in the DB Table Output operator.
| Output Table Type | Description |
|---|---|
Existing Table | No restrictions apply to Table Type. If Table Type is set to Duplicate Key:
|
Auto Created Table | In FineDataLink of versions earlier than V4.2.1.4, Table Type can only be set to Primary Key. In FineDataLink of V4.2.1.4 and later versions, Table Type can be set to Duplicate Key. You can set Table Type to Duplicate Key in Table Field Mapping. If Table Type is set to Duplicate Key:
|
Previous：PolarDB PostgreSQL Instruction
