4.1.1
Allowed you to set the log level for each pipeline task separately.
4.2.1.1
Changed Dirty Data Threshold to Table Dirty Data Threshold.
Allowed you to enable Retry After Task/Table Failure in Notification Content.
Allowed automatic retries when tables in a pipeline task encounter an exception.
4.2.1.2
The log level was set to INFO by default during pipeline task creation.
After establishing the mapping relationship between the source and target tables, you can configure execution-related settings of the pipeline task.
A synchronization task can maintain operation despite faults such as the mismatch between the field type or the field length and the primary key conflict after the threshold of the dirty data volume is set. The pipeline task is aborted automatically when the threshold is reached.
If you set Table Dirty Data Threshold to 1000 Row(s), the running pipeline task will be aborted when the number of dirty data records reaches 1000. The dirty data threshold limits the total number of dirty data records in a task since the task creation time.
A maximum of 100,000 dirty data rows can be tolerated. The dirty data counting is reset after the dirty data has been processed.
For details about dirty data processing, see Dirty Data Processing in Pipeline Task.
If a pipeline task or the synchronization of tables in the task fails due to network fluctuations or other reasons, the execution may be interrupted. Since the network is expected to recover after some time, you can set the number of retry times and the interval time in Retry after Failure to enable automatic reruns.
Retry Times
Interval
The default value is 2 (minutes) and the maximum value is 6 (minutes).
The following table describes the details.
Source network exception
Network connection failure (only network-related issues)
Retry After Failure is triggered automatically, and the logic is as follows.
If the full synchronization is not completed, the synchronization will restart from the beginning. If the full synchronization is completed, the incremental synchronization will resume from the last breakpoint. In other words, breakpoints do not exist in the full synchronization phase but only exist in the incremental synchronization phase.
The retry count is reset once the pipeline task reruns.
Configuration database exception
Configuration database read/write failure (caused by issues related to network, fields, permissions, and any other factors)
Message queue exception
Non-network-related issues in the source end (various source-end log anomalies)
Table Level
Dirty data generation
Database write failure due to any target-end anomalies
You can configure the notification for task exceptions, as shown in the following figure.
For details about Source Table Structure Change in Notification Content, see Data Pipeline - Synchronizing Source Table Structure Changes.
The failure notification of a single pipeline task is sent at most once within 10 seconds.
Notification Content
In V4.2.1.1 and later versions, you can enable the notification of retries caused by task/table failure.
The following table describes the reasons that trigger notifications.
Yes
The system retries the task and sends the notification.
The task is aborted.
No
The synchronization of tables with the dirty data volume below the threshold continues.
Dirty data threshold reached during the full synchronization phase
Dirty data threshold reached during the incremental synchronization phase
Platform/Email/SMS Description
When Notification Channel is set to SMS/Email/Platform, you can configure notification objects (including users, departments, and roles) based on the platform system. Ultimate objects would be the union of the selected.
When Notification Channel is set to SMS/Email, Custom Recipient and Platform User Group cannot be left empty at the same time. Besides, referencing parameters is not supported in Custom Recipient.
When you set Notification Channel to SMS/Email, select the user A in Platform User Group, and set Custom Recipient to the mobile number or the email address of the user A, the notification is sent via one channel only.
Dingtalk/Lark/WeCom Description
If you set Notification Channel to Client, you can set the notification channel to DingTalk/Lark/WeCom chatbots.
Webhook address for a WeCom chatbot: The following figure shows the steps for adding a chatbot. After the chatbot is successfully added, the Webhook address will be displayed on the prompt page.
You can set the log level for each pipeline task separately to meet different requirements such as log viewing, task debugging, and troubleshooting.
1. Output logs are those related to the reading and writing processes of pipeline tasks, including error and exception logs.
2. This function is disabled by default. When it is disabled, logs are recorded based on the global log level, which is set to WARN by default.
3. Configuration is only allowed when the pipeline task is not running and takes effect upon task startup.
4. You can select ERROR, WARN, or INFO in Log Level Setting.
Rank of log levels by severity (from highest to lowest): ERROR > WARN > INFO
Rank of log levels by detail (from simplest to most detailed): ERROR < WARN < INFO
Indicates an error that causes the service unavailable.
WARN
Indicates a warning of potential issues that do not cause the service unavailable, typically used for alerts.
INFO
Indicates general information about the running status or important events.
5. For a task configured with the log level, its log level setting takes precedence over the log level setting in Global Setting.
You can proceed with pipeline task O&M. For details, see Single Pipeline Task O&M and Batch Pipeline Task O&M.
滑鼠選中內容，快速回饋問題
滑鼠選中存在疑惑的內容，即可快速回饋問題，我們將會跟進處理。
不再提示
10s後關閉
Submitted successfully
Network busy