Pipeline Task Configuration - Pipeline Control

  • Last update: February 18, 2025
  • Overview

    FineDataLink VersionFunctional Change

    4.1.1

    Allowed you to set the log level for each pipeline task separately.

    4.2.1.1

    Changed Dirty Data Threshold to Table Dirty Data Threshold.

    Allowed you to enable Retry After Task/Table Failure in Notification Content.

    Allowed automatic retries when tables in a pipeline task encounter an exception.

    4.2.1.2

    The log level was set to INFO by default during pipeline task creation.

    After establishing the mapping relationship between the source and target tables, you can configure execution-related settings of the pipeline task.

    Procedure

    Table Dirty Data Threshold Setting

    A synchronization task can maintain operation despite faults such as the mismatch between the field type or the field length and the primary key conflict after the threshold of the dirty data volume is set. The pipeline task is aborted automatically when the threshold is reached.

    If you set Table Dirty Data Threshold to 1000 Row(s), the running pipeline task will be aborted when the number of dirty data records reaches 1000. The dirty data threshold limits the total number of dirty data records in a task since the task creation time.

    2025-02-14_14-35-13.png

    iconNote:
    1.  A maximum of 100,000 dirty data rows can be tolerated. The dirty data counting is reset after the dirty data has been processed.

    2. For details about dirty data processing, see Dirty Data Processing in Pipeline Task.

    Retry After Failure Setting

    If a pipeline task or the synchronization of tables in the task fails due to network fluctuations or other reasons, the execution may be interrupted. Since the network is expected to recover after some time, you can set the number of retry times and the interval time in Retry after Failure to enable automatic reruns.

    2025-02-14_14-35-54.png

    Setting ItemDescription

    Retry Times

    The default value is 3 and the maximum value is 10.

    Interval

    The default value is 2 (minutes) and the maximum value is 6 (minutes).

    The following table describes the details.

    LevelError DescriptionEnabling Retry After Failure
    Task Level

    Source network exception

    Network connection failure (only network-related issues)

    Retry After Failure is triggered automatically, and the logic is as follows.

    1. If the full synchronization is not completed, the synchronization will restart from the beginning. If the full synchronization is completed, the incremental synchronization will resume from the last breakpoint. In other words, breakpoints do not exist in the full synchronization phase but only exist in the incremental synchronization phase.

    2.  The retry count is reset once the pipeline task reruns.


    Configuration database exception

    Configuration database read/write failure (caused by issues related to network, fields, permissions, and any other factors)

    Message queue exception

    Network connection failure (only network-related issues)

    Non-network-related issues in the source end (various source-end log anomalies)

    The Retry After Failure function is not supported.
    Other unexpected exceptions

    Table Level

    Dirty data generation

    Database write failure due to any target-end anomalies

    Dirty data threshold reached during the full synchronization phase
    Dirty data threshold reached during the incremental synchronization phaseIn V4.2.1.1 and later versions, the automatic retry on dirty data will be triggered according to the configuration, and the table status will change to Dirty Data Processing in Progress.

    Result Notification Setting

    You can configure the notification for task exceptions, as shown in the following figure.

    iconNote:
    1. For details about Source Table Structure Change in Notification Content, see Data Pipeline - Synchronizing Source Table Structure Changes.

    2. The failure notification of a single pipeline task is sent at most once within 10 seconds.

    2025-02-14_14-36-53.png

    Notification Content

    In V4.2.1.1 and later versions, you can enable the notification of retries caused by task/table failure.

    The following table describes the reasons that trigger notifications.

    Level
    Error DescriptionTriggering Notification or Not
    Task Level

    Source network exception

    Network connection failure (only network-related issues)

    Yes

    The system retries the task and sends the notification.

    Configuration database exception

    Configuration database read/write failure (caused by issues related to network, fields, permissions, and any other factors)

    Message queue exception

    Network connection failure (only network-related issues)

    Non-network-related issues in the source end (various source-end log anomalies)

    Yes

    The task is aborted.

    Other unexpected exceptions

    Table Level

    Dirty data generation

    Database write failure due to any target-end anomalies

    No

    The synchronization of tables with the dirty data volume below the threshold continues.

    Dirty data threshold reached during the full synchronization phase

    No

    The task is aborted.

    Dirty data threshold reached during the incremental synchronization phase

    Yes

    The system retries the task and sends the notification.

    Platform/Email/SMS Description

    • When Notification Channel is set to SMS/Email/Platform, you can configure notification objects (including users, departments, and roles) based on the platform system. Ultimate objects would be the union of the selected.

    • When Notification Channel is set to SMS/Email, Custom Recipient and Platform User Group cannot be left empty at the same time. Besides, referencing parameters is not supported in Custom Recipient.

    • When you set Notification Channel to SMS/Email, select the user A in Platform User Group, and set Custom Recipient to the mobile number or the email address of the user A, the notification is sent via one channel only.

    Dingtalk/Lark/WeCom Description

    If you set Notification Channel to Client, you can set the notification channel to DingTalk/Lark/WeCom chatbots.

    iconNote:
    You are not allowed to enter multiple Webhook addresses for a single notification channel.

    2025-02-14_14-37-38.png

    Webhook address for a WeCom chatbot: The following figure shows the steps for adding a chatbot. After the chatbot is successfully added, the Webhook address will be displayed on the prompt page.

    2025-02-14_14-50-08.png

    Log Level Setting

    You can set the log level for each pipeline task separately to meet different requirements such as log viewing, task debugging, and troubleshooting.

    iconNote:
    In V4.2.1.2 and later versions, the log level is set to INFO by default during pipeline task creation.

    2025-02-14_14-38-36.png

    1. Output logs are those related to the reading and writing processes of pipeline tasks, including error and exception logs.

    2. This function is disabled by default. When it is disabled, logs are recorded based on the global log level, which is set to WARN by default.

    3. Configuration is only allowed when the pipeline task is not running and takes effect upon task startup.

    4. You can select ERROR, WARN, or INFO in Log Level Setting.

    • Rank of log levels by severity (from highest to lowest): ERROR > WARN > INFO

    • Rank of log levels by detail (from simplest to most detailed): ERROR < WARN < INFO

    Log LevelDescription
    ERROR

    Indicates an error that causes the service unavailable.

    WARN

    Indicates a warning of potential issues that do not cause the service unavailable, typically used for alerts.

    INFO

    Indicates general information about the running status or important events.

    5. For a task configured with the log level, its log level setting takes precedence over the log level setting in Global Setting.

    Subsequent Operations

    You can proceed with pipeline task O&M. For details, see Single Pipeline Task O&M and Batch Pipeline Task O&M.

    附件列表


    主题: Data Pipeline
    • Helpful
    • Not helpful
    • Only read

    滑鼠選中內容,快速回饋問題

    滑鼠選中存在疑惑的內容,即可快速回饋問題,我們將會跟進處理。

    不再提示

    4s后關閉

    Get
    Help
    Online Support
    Professional technical support is provided to quickly help you solve problems.
    Online support is available from 9:00-12:00 and 13:30-17:30 on weekdays.
    Page Feedback
    You can provide suggestions and feedback for the current web page.
    Pre-Sales Consultation
    Business Consultation
    Business: international@fanruan.com
    Support: support@fanruan.com
    Page Feedback
    *Problem Type
    Cannot be empty
    Problem Description
    0/1000
    Cannot be empty

    Submitted successfully

    Network busy