Overview
FineDataLink Version | Functional Change |
---|---|
4.1.1 | Allowed you to set the log level for each pipeline task separately. |
4.2.1.1 | Changed Dirty Data Threshold to Table Dirty Data Threshold. Allowed you to enable Retry After Task/Table Failure in Notification Content. Allowed automatic retries when tables in a pipeline task encounter an exception. |
4.2.1.2 | The log level was set to INFO by default during pipeline task creation. |
After establishing the mapping relationship between the source and target tables, you can configure execution-related settings of the pipeline task.
Procedure
Table Dirty Data Threshold Setting
A synchronization task can maintain operation despite faults such as the mismatch between the field type or the field length and the primary key conflict after the threshold of the dirty data volume is set. The pipeline task is aborted automatically when the threshold is reached.
If you set Table Dirty Data Threshold to 1000 Row(s), the running pipeline task will be aborted when the number of dirty data records reaches 1000. The dirty data threshold limits the total number of dirty data records in a task since the task creation time.

A maximum of 100,000 dirty data rows can be tolerated. The dirty data counting is reset after the dirty data has been processed.
For details about dirty data processing, see Dirty Data Processing in Pipeline Task.
Retry After Failure Setting
If a pipeline task or the synchronization of tables in the task fails due to network fluctuations or other reasons, the execution may be interrupted. Since the network is expected to recover after some time, you can set the number of retry times and the interval time in Retry after Failure to enable automatic reruns.
Setting Item | Description |
---|---|
Retry Times | The default value is 3 and the maximum value is 10. |
Interval | The default value is 2 (minutes) and the maximum value is 6 (minutes). |
The following table describes the details.
Level | Error Description | Enabling Retry After Failure |
---|---|---|
Task Level | Source network exception Network connection failure (only network-related issues) | Retry After Failure is triggered automatically, and the logic is as follows.
|
Configuration database exception Configuration database read/write failure (caused by issues related to network, fields, permissions, and any other factors) | ||
Message queue exception Network connection failure (only network-related issues) | ||
Non-network-related issues in the source end (various source-end log anomalies) | The Retry After Failure function is not supported. | |
Other unexpected exceptions | ||
Table Level | Dirty data generation Database write failure due to any target-end anomalies | |
Dirty data threshold reached during the full synchronization phase | ||
Dirty data threshold reached during the incremental synchronization phase | In V4.2.1.1 and later versions, the automatic retry on dirty data will be triggered according to the configuration, and the table status will change to Dirty Data Processing in Progress. |
Result Notification Setting
You can configure the notification for task exceptions, as shown in the following figure.

For details about Source Table Structure Change in Notification Content, see Data Pipeline - Synchronizing Source Table Structure Changes.
The failure notification of a single pipeline task is sent at most once within 10 seconds.
Notification Content
In V4.2.1.1 and later versions, you can enable the notification of retries caused by task/table failure.
The following table describes the reasons that trigger notifications.
Level | Error Description | Triggering Notification or Not |
---|---|---|
Task Level | Source network exception Network connection failure (only network-related issues) | Yes The system retries the task and sends the notification. |
Configuration database exception Configuration database read/write failure (caused by issues related to network, fields, permissions, and any other factors) | ||
Message queue exception Network connection failure (only network-related issues) | ||
Non-network-related issues in the source end (various source-end log anomalies) | Yes The task is aborted. | |
Other unexpected exceptions | ||
Table Level | Dirty data generation Database write failure due to any target-end anomalies | No The synchronization of tables with the dirty data volume below the threshold continues. |
Dirty data threshold reached during the full synchronization phase | No The task is aborted. | |
Dirty data threshold reached during the incremental synchronization phase | Yes The system retries the task and sends the notification. |
Platform/Email/SMS Description
When Notification Channel is set to SMS/Email/Platform, you can configure notification objects (including users, departments, and roles) based on the platform system. Ultimate objects would be the union of the selected.
When Notification Channel is set to SMS/Email, Custom Recipient and Platform User Group cannot be left empty at the same time. Besides, referencing parameters is not supported in Custom Recipient.
When you set Notification Channel to SMS/Email, select the user A in Platform User Group, and set Custom Recipient to the mobile number or the email address of the user A, the notification is sent via one channel only.
Dingtalk/Lark/WeCom Description
If you set Notification Channel to Client, you can set the notification channel to DingTalk/Lark/WeCom chatbots.

Webhook address for a WeCom chatbot: The following figure shows the steps for adding a chatbot. After the chatbot is successfully added, the Webhook address will be displayed on the prompt page.
Log Level Setting
You can set the log level for each pipeline task separately to meet different requirements such as log viewing, task debugging, and troubleshooting.

1. Output logs are those related to the reading and writing processes of pipeline tasks, including error and exception logs.
2. This function is disabled by default. When it is disabled, logs are recorded based on the global log level, which is set to WARN by default.
3. Configuration is only allowed when the pipeline task is not running and takes effect upon task startup.
4. You can select ERROR, WARN, or INFO in Log Level Setting.
Rank of log levels by severity (from highest to lowest): ERROR > WARN > INFO
Rank of log levels by detail (from simplest to most detailed): ERROR < WARN < INFO
Log Level | Description |
---|---|
ERROR | Indicates an error that causes the service unavailable. |
WARN | Indicates a warning of potential issues that do not cause the service unavailable, typically used for alerts. |
INFO | Indicates general information about the running status or important events. |
5. For a task configured with the log level, its log level setting takes precedence over the log level setting in Global Setting.
Subsequent Operations
You can proceed with pipeline task O&M. For details, see Single Pipeline Task O&M and Batch Pipeline Task O&M.