Overview
Version
FineDataLink Version | Functional Change |
---|---|
4.1.11.2 | Pipeline Task supported the configuration of partitions and distribution logic for the target table when the target table was in the YMatrix database and set to Auto Created Table. For details, see Partition Table Creation and Data Reading/Writing. |
4.1.11.5 | Filled in the target table description with the source table description automatically when you selected Auto Created Table. For details, see the "Table Description and Field Remark" section of this document. |
Function Description
In the Table Field Mapping step, you can map source fields to target fields, as shown in the following figure.
Procedure
Target Table Creation
The target table name cannot contain special characters. You can create a table or select an existing table as the target table.
Auto Created Table
1. You can select Auto Created Table, which will create a target table in the target database.

2. The following describes the Manual Table Creation function.
Scenario one: You can copy the statement in Manual Table Creation, and paste and execute it in the target database to create a table using the statement.
Scenario two: You can modify the field name and data type and define table attributes in the Manual Table Creation window. For details, see Manual Table Creation.
3. The following describes the special setting of some databases in Pipeline Task.
You need to select the table type when writing data into a Doris database. For details, see Doris Instruction.
You need to select the table type when writing data into a StarRocks database. For details, see StarRocks Instruction.
Existing Table
You can select Existing Table, which will synchronize data selected in the pipeline task to an existing table in real time, as shown in the following figure.

1. If you selected Logical Deletion at Target End and Mark Timestamp During Synchronization in the Target Selection step when configuring a pipeline task and the target table does not contain the _fdl_update_timestamp and the _fdl_marked_deleted fields, the system will automatically create the fields when the task is running.
2. In versions from 4.0.30 to 4.1.5.2, when you select Incremental Synchronization Only in the Source Selection step, the target table defaults to Existing Table that has the same name as the source table. You can change the target table from Existing Table to Auto Created Table. In versions 4.1.5.2 and later, when you select Incremental Synchronization Only in the Source Selection step, the target table defaults to Auto Created Table.
Primary Key Setting
You can replace the physical primary key with the logical primary key for real-time data synchronization.

1. The existing physical primary key in the target table is preferred during synchronization. The logical primary key you set will be used if no such physical primary key exists.
2. If you set a physical primary key for the target table of a task that has been configured with the logical primary key, the physical primary key of the target table, instead of the logical primary key, will be displayed as the primary key when you view the field mapping.
Target Table Type | Primary Key Status of Target Table | Source Table | Target Table | ||
---|---|---|---|---|---|
Auto Created Table | The target database supports the physical primary key. | The source table has the physical primary key. | The physical primary key of the source table is mapped to the target table automatically. | ||
The source table has no physical primary key. | You can set the physical primary key. | You have set the physical primary key. | 1. The system sets the primary key of the target table to the specified field during automatic table creation. 2. The physical primary key of the target table is used as a comparison field for updating and deleting data during synchronization. | ||
You have not set the physical primary key. (Synchronization Without Primary Key is enabled.) | 1. The system does not set the primary key during automatic table creation. 2. All usable fields in the target table are set to logical primary keys as comparison fields for updating and deleting data during synchronization. | ||||
The target database does not support the physical primary key and supports the logical primary key only. | The source table has the physical primary key. | The physical primary key of the source table is mapped to the target table automatically. | You have set the logical primary key. ![]() | The system does not set the primary key during automatic table creation. The set logical primary key is used as a comparison field for updating and deleting data during synchronization. | |
The source table has no physical primary key. | You can set the logical primary key. | You have not set the logical primary key. (Synchronization Without Primary Key is enabled.) | The system does not set the primary key during automatic table creation. All usable fields in the target table are set to logical primary keys as comparison fields for updating and deleting data during synchronization. | ||
Existing Table | The target table has the physical primary key. ![]() | / | The existing physical primary key is used by default and cannot be modified. The physical primary key of the target table is used as a comparison field for updating and deleting data during synchronization. | ||
The target table has no physical primary key. ![]() | The source table has the physical primary key. | The physical primary key of the source table is mapped to the target table automatically. | |||
The source table has no physical primary key. | You can set the logical primary key. | You have set the logical primary key. | The set logical primary key is used as a comparison field for updating and deleting data during synchronization. | ||
You have not set the primary key. (Synchronization Without Primary Key is enabled.) | All usable fields in the target table are set to logical primary keys as comparison fields for updating and deleting data during synchronization. |
1. The system automatically identifies the existing physical primary key of the source table if you select Auto Created Table.
2. The system automatically identifies the primary key of the existing table if you select Existing Table.
3. You can manually set the primary key of the target table.
Partition Key Setting
You can specify the partition (for PostgreSQL, Greenplum, Gauss 200, or YMatrix database table) and distribution logic (for Greenplum, Gauss 200, or YMatrix database table) when you select Auto Created Table as the target table. For details, see Partition Table Creation and Data Reading/Writing.
Field Mapping
You can adjust the field mapping method, modify the field data type, the field name, and the field order of the target table, unmap fields, and filter mapped and unmapped fields, as shown in the following figure.
Modifying Table Names and Table Creation Methods in Batches
1. To prefix or suffix table names in batches, you can click Batch Modify on the upper right corner, select the tables to be modified, and add the prefix or the suffix, as shown in the following figure.
The target table name is prefixed with ods_, as shown in the following figure.
2. You can modify the table creation method of tables in batches.

Filter Introduction
1. Click the filter icon beside the search bar. You can filter the source table to be synchronized by the configuration status, the primary key status, and the creation method of the target table.
If you have enabled Synchronization Without Primary Key at the Target Selection step and set no primary key, all usable fields in the table will be set to logical primary keys during the task running.
2. Click the filter icon in the upper right corner of the field mapping area. You can filter fields by error status and mapping status.
Errors of target table fields include empty field names and data types and others.
Table Description and Field Remark
You can synchronize the description and the field remark of the source table to the target table. If the target table is set to Auto Created Table, you can customize its description and field remarks.
1. The field remark and the description of the source table are displayed in the Source area during field mapping.
2. You can customize the field remark and the description of the target table when selecting Auto Created Table as the target table.
The target table description is filled with the source table description automatically when you select Auto Created Table, as shown in the following figure.
3. The field remark and the description of the target table are displayed when you select Existing Table as the target table.
Note that:
1. Databases that do not support Table Description include ClickHouse, Hive, Impala, Transwarp Inceptor, Informix, MaxCompute, SQLite, and StarRocks.
2. Databases that do not support the reading or writing of the field remark include Hive, Impala, Transwarp Inceptor, Informix, MaxCompute, SQLite, and StarRocks.
3. If the description and the field remark of the source table change:
Synchronize Source Table Structure Change is disabled:
FineDataLink only synchronizes the field remark during the first synchronization. Subsequent changes to the field remark of the source table are not synchronized to the target table.
Synchronize Source Table Structure Change is enabled:
Changes to the field remark (including additions, deletions, and modifications) of the source table are synchronized to the target table. The remark of new fields is also synchronized.
FineDataLink only synchronizes the table description during the first synchronization, no matter whether Synchronize Source Table Structure Change is enabled or not. Subsequent changes to the description of the source table are not synchronized to the target table.
Differences Between the Group Table and the Ordinary Table
The above sections in the "Procedure" chapter of this document also apply to group tables. This section describes the differences between the group table and the ordinary table.
1. If the data type of a field in the group table differs across subtables, the system will select a type compatible with all data types of this field as the data type.
2. The system adds a _fdl_src_schema_table field to the target table automatically when the source table is a group table to record schema names and table names of subtables of the group table. The field value is in the form of Schema name.Table name.
3. The configuration of Physical Primary Key or Not Null of a field in the group table is retained only when the configuration of Physical Primary Key or Not Null of this field is the same in all sub-tables. Otherwise, the two items of this field are not configured.
Notes
1. Use the CollapsingMergeTree table engine when writing data to the target table in the ClickHouse database. For details, see ClickHouse Instruction.
2. FineDataLink's Pipeline Task supports limited field data types for the Oracle data source. For details, see Oracle Instruction.
Subsequent Operations
For details, see Pipeline Task Configuration - Pipeline Control.