Overview of Data Synchronization

Application Scenario

You can use the Data Synchronization node to synchronize data in the following scenarios:

• You want to synchronize data that needs no complex processing quickly.

• You want to synchronize data from tables with large data volumes (where a single table contains over 10 million records) or synchronize data to databases with strong computing power.

Function Description

Support for Multiple Types of Data Sources

You can synchronize data from one endpoint to any other endpoint. For details, see Description of Scheduled Task Capabilities.

For example, you can synchronize Jodoo data to databases or save database data as file data.

Fast Synchronization of Data Tables

You can extract data from the source end and write it into the target database by simply configuring Data Source, Data Destination and Mapping, and Write Method.

• You are a data development user of FineDataLink. For details, see User Management.

• You have permission on Data Development. For details, see Data Platform Use Permission and Scheduled Task Management Permission.

• You have configured the data connection of the data source in FineDataLink. For details, see Data Source Creation and Management.

• Data Synchronization supports the synchronization of structured, semi-structured, and unstructured data (such as data in OSS objects and text files, which must be converted into structured data). Data Synchronization can be used to synchronize only the data that can be abstracted to two-dimensional logical tables and cannot synchronize unstructured data that cannot be converted to structured data, such as data in MP3 files that are stored in OSS.

• Data Synchronization is used for timed data synchronization. For details about real-time data synchronization, see Overview of Data Pipeline.

• You can configure the write method, the primary key mapping, and the strategy for primary key conflict to ensure data uniqueness after synchronization. For details, see Overview of Data Synchronization Method.

• Some data sources provide limited support for Data Development. For details, see Functional Limitations of Different Data Sources.

You can use the Data Synchronization node to extract data from the source end to the target end and perform basic data processing. The setting page is shown in the following figure.

Note:

If the data to be synchronized needs complex cleaning and processing, you are advised to use the Data Transformation node.

Procedure	Description
Step 1: Setting the Data Source (Required)	1. Data synchronization between more than 30 data source types (including relational, non-relational, API, and file data) is currently supported. By defining Data Source and Data Destination, you can achieve data transfer between any structured and semi-structured data. For details about the supported data sources, see Data Sources Supported by FineDataLink. 2. For details about each setting item on the Data Source tab page, see Data Synchronization - Data Source.
Step 2: Setting the Data Destination and Mapping (Required)	For details about each setting item, see Data Synchronization - Data Destination and Mapping. Note: In FineDataLink V4.0.18, Comparison Field is removed. You can perform incremental data addition, deletion, and update operations by using a combination of Data Comparison and DB Table Output.
Step 3: Setting the Write Method (Required)	For details about each setting item, see Data Synchronization - Write Method.

In V4.0.28 and later versions, the following scenarios (where neither the source table nor the target table uses Jodoo data) can be implemented by using the Data Synchronization node:

• Add/modify/delete the filtered data without the identifier field. (Only one operation type can be selected when the identifier field is empty.)

• Add/modify/delete data with the identifier field and its value.

For details, see Data Synchronization - Add/Modify/Delete Data Based on Identifier Field.