Execution Details of Real-Time Pipeline Tasks

Version

Application Scenario

After configuring data pipeline tasks, you, the IT personnel, may need to monitor the synchronization status and performance of the tasks and handle exceptions.

Function Description

You can click a started pipeline task to view its configuration, synchronization details, and execution logs, as shown in the following figure.

• You can add, edit, and delete tables, as well as modify Task Control settings. For details, see Real-Time Pipeline Task Management.

• You can view real-time synchronization status by time/table dimension.

• You can view historical execution logs and query and filter logs as needed.

Note:

For details about handling dirty data generated in pipeline tasks, see Real-Time Pipeline Task - Dirty Data Processing.

For FineDataLink of V4.2.11.3 and later versions, see Editing Pipeline Tasks.

Viewing Real-Time Synchronization Status by Time Dimension

Click Pipeline Activity to view the synchronization details of the current task.

Note:

On Real-Time Statistics and Historical Statistics tab pages, the synchronization object is divided into group tables and ordinary tables. You can expand a group table to view its subtables.

Real-Time Statistics

On the Real-Time Statistics tab page, three indicator cards are displayed, including Pending Data, Total Written, and Dirty Data.

• Pending Data: the total backlog of unconsumed messages in the CDC phase across all tables in the task message queue

• Total Written: the total number of written messages from all tables in the task, where the number of written rows in the full and incremental synchronization phases (with the number of inserted, updated, and deleted rows) is displayed separately

• Dirty Data: the total number of dirty data records generated by all tables in the task

Note:

If resynchronization is used, the read and write counts of a single table are recalculated. The recynchronized data volume is not counted in Real-Time Statistics, but is counted in Historical Statistics.

For group tables, the dirty data count, pending data count, and total written count are the sum of the corresponding indicator values of the tables within the group.

For example, a group table A contains subtables A1 and A2:

• Dirty data count of the group table A = Count of dirty data from A1 generated during writes to A + Count of dirty data from A2 generated during writes to A

• Pending data count of the group table A = Pending data count of A1 + Pending data count of A2

• Total written data count of the group table A = Count of written data (inserted, updated, and deleted data) from A1 + Count of written data (inserted, updated, and deleted data) from A2

Historical Statistics

1. Since FineDataLink V4.1.2, you can select Last 2 Hour(s), Last 24 Hour(s), Last 3 Day(s), Last 7 Day(s), and Last 15 Day(s) to view the real-time synchronization statistics accordingly.

2. Pending Data and Total Written are displayed, as shown in the following figure.

• Pending Data: the total backlog of unconsumed messages in the CDC phase across all tables

• Total Written: the total number of written rows, where the number of written rows in the full and incremental synchronization phases (with the number of inserted, updated, and deleted rows) is displayed separately

You can click the line/column chart to view it enlarged, with a minimum granularity of one minute on the horizontal axis, as shown in the following figure.

• The Write Latency indicator appears only after you select a source table in the Read and Write Statistics window (added since FineDataLink V4.2.14.4).

• You can view the write latency changes of a specified table within the specified time period.

Viewing Synchronization Status by Table Dimension

You can view the synchronization status of all source tables in Pipeline Activity, as shown in the following figure.

1. The following table describes the columns.

Column Name	Description
Source Table	Source table name Hover over the source table name to view the source database and schema.
Target Table	Target table name, which is available starting from FineDataLink V4.2.6.3 Hover over the target table name to view the target database and schema.
Synchronization Status	Current synchronization phase of the table, such as Incremental Synchronization in Progress and Full Synchronization in Progress Starting from FineDataLink V4.2.1.1, new table statuses, including Pending Initialization, Initializing, Pending Full Synchronization, Pending Incremental Synchronization, Dirty Data Processing in Progress, and Aborted, have been added. For details, see Real-Time Pipeline Task Status and Table Status.
Latest Message Read Time	Sorting is supported.
Pending Data	The amount of data pending synchronization from a single table Sorting is supported.
Latest Message Write Time	The latest write time of a single table within the task Sorting is supported.
Written Data	Total number of messages written for a single table within the task (introduced since V4.2.6.3), which is the sum of the number of written rows during the full and incremental synchronization phase The number of written rows in the full and incremental (with the number of inserted, updated, and deleted rows) synchronization phases is displayed separately.
Dirty Data	1. The amount of dirty data generated by a single table within the task Sorting and filtering are supported. 2. Click the dirty data count to view details, where you can filter dirty data records by error type and keyword, and sort them by time. Note: For details about dirty data processing, see Real-Time Pipeline Task - Dirty Data Processing. Click the  icon to export dirty data. Note: If a record with a primary key A is updated to a new value B, two dirty data records will be generated upon update failure, namely, delete the record with a primary key value A and insert the record with a primary key value B. The data will be exported according to the structure of the target table configured in Field Mapping.
Write Latency	The time difference between the time reading the message from the source table and the time writing the message to the target table Sorting is supported.

2. You can customize the table header, as shown in the following figure.

3. You can click the  icon in Historical Statistics to view the execution logs of the table, as shown in the following figure.

4. You can resynchronize tables and pause/start table synchronization in Real-Time Statistics. For details, see Real-Time Pipeline Task Management.

Visual Log

1. You can click Execution Log to view the historical execution logs of the current pipeline task. Details include Time, Level, Classification, and Description, as shown in the following figure.

2. You can also query and filter logs, as shown in the following figure.

Note:

You can choose to view task logs or table logs, and filter logs by log classification and generation time.

Click the  icon. The filter settings are shown in the following figure.

Log Level

Logs are divided into four levels: BASIC INFO, INFO, WARN, and ERROR.

BASIC INFO is the basic level.

Click Details next to each log to view the specific description, as shown in the following figure.