Data Synchronization - Data Source

Version

Function Description

This document introduces configuration items of setting the data source in the Data Synchronization node or using the DB Table Input operator in the Data Transformation node.

The following figure displays the Data Source tab page of the Data Synchronization node.

Use the Generate Data Transformation function to implement complex processing and transformation of the fetched data, such as column-to-row conversion, data association, and JSON parsing.

For details, see the chapter "Types of Data Sources Supported by Scheduled Task" of "Data Sources Supported by FineDataLink."

The drop-down list of Data Source only contains the data sources connected via data connections on which you have Use permission.

If you select DB Table Input in Data Source, you can select SQL, Table Selection, or Stored Procedure as Configuration Method.

SQL

Query and fetch data from the source database tables through SQL statements. Referencing parameters is supported. For details about parameters, see Overview of Parameter.

You can select the specified database, search for the required data table, and click the table name for it to be written in the SQL statement into the SQL input box on the right, as shown in the following figure.

Notes:

If you have Use permission on multiple databases connected via the selected data connection, specify the table in the format of Database.Table name in the SQL statement.

Table Selection

The following figure displays the effect.

The following operators/nodes support the Table Selection function if you select DB Table Input in Data Source: DB Table Input, Data Synchronization, Parameter Assignment, and Global Parameter.

Source Table

Select the source table, which cannot be empty.

In V4.1.13.2 and later versions, partial field selection is supported, as shown in the following figure.

The Partial Field Selection function is explained as follows.

1. The Select All setting does not take effect on subsequent new fields in the table.

2. Field auto-completion is also supported for unselected source table fields in the Data Filtering area on the Data Source tab page.

3. You can unselect fields in two ways, as shown in the following figure.

4. The source end of field mapping on the Data Destination and Mapping tab page only displays the selected fields. Filtering and re-obtaining are only valid for the selected fields.

5. Only the selected fields will be displayed on the Primary Key Mapping configuration page when you configure Write Method.

6. DDL description:

• If the Partial Field Selection function is used, the DDL Change Detection function on the Data Destination and Mapping tab page will not be displayed.

• DDL Change Detection monitors the structure changes of selected fields. The Update Field Mapping function is not affected.

Table Description

Table Description will be displayed if you select DB Table Input in Data Source and Table Selection as Configuration Method in the DB Table Input operator or the Data Synchronization and Parameter Assignment nodes. If you select Auto Created Table in Target Table, the table description of the target table defaults to that of the source table.

Notes:

• Databases that do not support Table Description include ClickHouse, Hive, Impala, Transwarp Inceptor, Informix, MaxCompute, SQLite, and StarRocks.

• For details, see Data Synchronization - Data Destination and Mapping.

Data Filtering

• The content can be empty.

• Data in the source table can be filtered. The content is the WHERE conditional statement written in the syntax of the selected database. (You do not need to fill in the WHERE keyword.)

• Referencing various parameters is supported.

• Linking parameter values and fields in the data table is supported.

• LIMIT statements are not supported.

Parallel Read

When the volume of data is large, enable Parallel Read to increase data reading speed.

Notes:

If Parallel Read is enabled, appending data to existing tables and setting a logical primary key may result in data duplication.

Setting Item	Description
Parallel Read	It is not enabled by default. It is effective only when the following three conditions are met at the same time: Data source: Currently, this function only supports MySQL, Oracle, SQL Server, and Vertica databases. Data destination: databases of the JDBC type Operator/node: the Data Synchronization node, and the DB Table Input and DB Table Output operators of the Data Transformation node
Split Key	Select the fields in the table from the drop-down list, and the data will be separated according to the split keys. You are advised to use the primary key field. The field type of the split key needs to be Value. The split key cannot be empty.
Read Concurrency	It cannot be empty and can be modified. During actual execution, the concurrency will not exceed the number of CPU cores. Notes: The concurrency only represents the number of concurrently executed threads, instead of the actual number of split tasks. On the Statistics tab page, the concurrency may be 2, and the actual number of split tasks may be 3.

Stored Procedure

For details, see Scheduled Task Calling Database Stored Procedures.