Task Control - Fault Tolerance Mechanism

Version

Function Description

You can configure Timeout Limit, Retry After Failure, and Dirty Data Tolerance on the Fault Tolerance Mechanism tab page, as shown in the following figure.

Note:

The Result Notification configuration of a scheduled task takes effect only when you publish the task to Production Mode.

A task interrupted due to network fluctuations or other reasons can be executed successfully if you rerun it after a while.To prevent such task interruption, you can configure the number of retries and the interval between retries in Retry After Failure to automatically rerun the task upon failure.

The Retry After Failure function is disabled by default and needs to be enabled manually.

Note:

1. The logic of Retry After Failure: The setting takes effect for the failed node only.

2. Situations like network interruptions may cause database disconnection and partial data appending. In such cases, rerunning the task may result in duplicate data appending or errors due to primary key conflicts.

3. If Retry After Failure is enabled for the task invoked using the Invocation Task node, the entire subtask will be re-executed, rather than the failed node in the subtask only.

4. If a scheduled task contains a Loop Container node with Retry After Failure enabled:

For FineDataLink of versions before V4.1.0:

• If the failed node is within Loop Container, the entire Loop Container node will be rerun.

• If the Loop Container node is set to run five times but a node within it fails during the second iteration, the entire node will rerun from the first iteration.

In FineDataLink of V4.0.29 and later versions:

• When a node in Loop Container fails, the system will immediately retry the failed node using the parameters of the current loop instead of retrying the entire Loop Container node.

5. If Retry After Failure is enabled for a task with the number of retries set to 3, the task will only be considered failed after all three retries have failed.

The execution judgment on the connector specifies that the Notification node is executed only if the Data Synchronization node fails, as shown in the following figure.If the number of retries is set to 3 and the Data Synchronization node fails, the Notification node will be executed when all three retries of the Data Synchronization node fail.

A task that has been running for a long time may experience exceptions and occupy resources.To address this issue, you can configure timeout limits in Timeout Limit for tasks to forcibly terminate running tasks that run longer than the set duration.

The Timeout Limit function is disabled by default and needs to be enabled manually.The default value is one hour, and the maximum value is 48 hours.

You can set Dirty Data Threshold to enhance the fault tolerance of tasks.The scheduled task continues running despite dirty data and does not trigger the error until the set limit of dirty data is reached.

Dirty Data Definition:

For details, see Dirty Data.

Note:

For running scheduled tasks, the cumulative dirty data counted by Dirty Data Threshold does not include the dirty data generated due to data processing exceptions (for example, JSON parsing failure) during transformation.

Function Description:

1. The Dirty Data Tolerance button is disabled by default. The default value of Dirty Data Threshold is 1000, with an input range of 1 to 10,000.

2. If Dirty Data Threshold is set, erroneous data in all output components will be routed to the dirty data queue and recorded as dirty data generated in the corresponding operator/node.

An output component must meet all the following conditions:

Note:

For Redshift (with high-speed loading enabled), StarRocks, and Doris databases, the data writing is subject to the threshold setting, but FineDataLink cannot track the reasons for dirty data statistically.

• It is a Data Synchronization node or an output operator in the Data Transformation node.

• DB Table Output is selected as the output end, and the data destination is not a Hive (HDFS), GP (with Parallel Loading enabled), or Transwarp database.

3. The Dirty Data Threshold configuration applies to all output components in the task. For example, if you set Dirty Data Threshold to 1000, all output components in the task will be subject to the same node-level dirty data threshold of 1000."

If dirty data is generated during task execution:

1. The Statistics tab page in the task log display area shows dirty data information. You can click the number of dirty data rows to view the row count of dirty data caused by different reasons.

2. You can choose O&M Center > Running Record and click the View Details button on the right side of a task instance to retry the task and view retry records.For details about the task retry function and retry scenarios, see Task Retry.

3. If the number of generated dirty data records does not exceed the dirty data threshold, the scheduled task will continue running despite dirty data and be labeled Successful in Running Status if it is executed successfully.