Real-Time Capture Task O&M

Version

FineDataLink Version	Functional Change
4.2.4.3	Added a Real-Time Capture Task function.
4.2.6.3	1. Added a Dependency tab page to display the list of pipeline tasks and real-time tasks that depend on this real-time capture task. 2. Displayed Log Parsing Latency when the source-end database was Oracle, MySQL, or SQL Server. 3. Added Real-Time Statistics and Historical Statistics tabs to the Table Being Parsed tab page. 4. Allowed viewing the execution status of all data capture tasks.
4.2.10.4	Displayed the Topic's used storage, available storage, and earliest data timestamp in the Topic when you hovered over the  icon beside the source table name.
4.2.13.1	Made the real-time capture task to periodically check the following items when the source end of the real-time pipeline tasks and real-time tasks was SQL Server: The status of Agent and Capture jobs, with prompts in Execution Log The source table structure and CDC table structure (excluding field types) of tables currently being captured (If inconsistencies were found in column count or field names between source and CDC tables, the capture task printed task-level WARN logs.)
4.2.14.1	1. Displayed the data connection URL on the details page of the real-time capture task. 2. Displayed the Delete button in the upper-right corner of the real-time capture task. 3. Allowed manually pausing real-time capture tasks. 4. Optimized the status of real-time capture tasks. 5. Allowed removing tables from single/multiple real-time capture tasks. 6. Allowed viewing backfill records and canceling backfill operations.
4.2.14.4	Added the Log Parsing Latency indicator to the Historical Statistics tab page.

Function Description

After a real-time capture task is automatically created, you can view its execution details under System Management > Data Connection > Real-Time Capture Task, as shown in the following figure.

Filtering Real-Time Capture Tasks

You can filter real-time capture tasks by connection name, task name, name of the table being parsed, and task status, as shown in the following figure.

The following table describes the task status.

Renaming/Deleting/Pausing Real-Time Capture Tasks

Renaming

You can rename real-time capture tasks, as shown in the following figure.

Deletion

Real-time capture tasks in the Stopped or Runtime Error status can be deleted, as shown in the following figure.

Application scenarios of the delete operation include:

• Existing capture tasks are no longer needed.

• Checkpoints of existing capture tasks are not required.

• Logs for capture task checkpoints are lost.

You can reset the capture task for a data connection by deleting the capture task.

Deletion logic:

Deleting a capture task removes all related information, including capture task checkpoints and logs, all Kafka Topics in this capture task, and all data within those Kafka Topics (including old Topics generated from backfill operations).

Pause

This operation has been supported since V4.2.14.1.

Real-time capture tasks in the Running status can be paused, as shown in the following figure.

Application scenarios of the pause operation include:

If real-time capture significantly impacts the database, you can immediately stop the capture task using the Pause button.

Pause logic:

Note:

Real-time pipeline task/Real-time task exceptions caused by a manual pause of capture tasks do not trigger the retry mechanism for real-time pipeline tasks/real-time tasks.

• When you click Pause, the capture task enters the stopping process, stops capturing data from all tables in this task, switches its status from Running to Stopping, and pauses all backfill threads.

• Running real-time tasks using this capture task will abort with an error, indicating that the capture task is manually paused and the synchronization of corresponding tables is aborted.

• Running real-time pipeline tasks using this capture task will abort with an error, indicating that the capture task is manually paused and the real-time synchronization is aborted.

Backfill

Backfill logic:

Operations such as adding tables to real-time pipeline tasks/real-time tasks, synchronizing data earlier than the earliest data in the real-time capture task, or resuming synchronization for historical tables are considered backfill operations.

In other words, backfill refers to the process where the current capture task, having already parsed up to the latest timestamp, is required to re-parse previous logs for real-time pipeline tasks or real-time tasks, resulting in multiple threads parsing logs simultaneously in a short period.

Note:

If an error occurs in any table, the entire backfill process will abort.

1. When a table is added, or a historical table resumes capturing data, the table is displayed in the real-time capture task with the status Parsing, or Queuing when queued.

2. From FineDataLink V4.2.14.1, if the capture task has historically executed a backfill or is currently executing a backfill, you can click the View Backfill Record button on the real-time capture task details page to view the backfill records, as shown in the following figure.

The following figure shows the View Backfill Record page.

3. Canceling accidentally started backfill tasks that take a long time is supported from FineDataLink V4.2.14.1.

A Cancel button is displayed when the backfill task is in the Queuing or Backfilling status. 

After the backfill task is canceled:

• The Kafka Topic related to this sub-collector and the data in it are deleted.

• Running real-time tasks that depend on this backfill task abort with an error, indicating that backfill for Table name in the capture task is manually canceled, stopping the synchronization.

• Running real-time pipeline tasks that depend on this backfill task abort with an error, indicating that the backfill for Table name in the capture task is manually canceled, stopping the synchronization, while the synchronization of other tables for which the backfill is not canceled continues normally.

Select a single real-time capture task to view its basic information, tables being parsed, dependencies, and execution logs, as shown in the following figure.

Basic Information

You can view the data connection name, read method, task creation time, latest read message time, log parsing latency (added since V4.2.6.3), and data connection URL (added since V4.2.14.1) of the real-time capture task, as shown in the following figure.

If the source-end database is Oracle, MySQL, or SQL Server, Log Parsing Latency is displayed.

Table Being Parsed

Note:

The Real-Time Statistics and Historical Statistics tabs have been added since V4.2.6.3.

Real-Time Statistics

This tab page displays all tables currently being captured by this capture task. The tab page is shown in the following figure.

1. The following table explains the table columns and the Total Read indicator.

Item	Description
Indicator: Total Read (Row)	It shows the total message volume read by all tables in the task (increment only). Read messages are classified into inserted, updated, and deleted ones and counted separately.  Total read volume = Inserted volume + Updated volume + Deleted volume
Table column	Source Table Name: From FineDataLink V4.2.10.4, hovering over the  icon beside the source table name displays the Topic's used storage, available storage, and the earliest data timestamp in the Topic.
	Task: It lists the specific real-time pipeline task or real-time task that the table belongs to.
	Capture Start Point: It displays the time point when the current collector of this table starts capturing data.
	Latest Read Message Time: /
	Read Volume: It shows the total volume of messages read by a single table in the task (increment only), classified into inserted, updated, and deleted records.
	Creation Time: It displays the time when the table capture is first created.
	Status: 1. Backfilling: The backfill is in process. When the system backfills earlier data for a table, the table may be captured by both the main collector and the sub-collector simultaneously. In this case, the table status is Parsing + Queuing. 2. Parsing (Running): The table is being captured. 3. Stopped: Task-level error Task-level pause 4. Runtime Error: The table encounters a table-level error (for example, table-level DDL or DML parsing exceptions) during startup or running, and the capture stops. From FineDataLink V4.2.14.1, error details are displayed.

2. From FineDataLink V4.2.14.1, batch deletion of tables in a real-time capture task is supported, as shown in the following figure.

3. From FineDataLink V4.2.14.1, prompts are provided for unused tables in the capture tasks, as shown in the following figure.

Historical Statistics

The tab page is shown in the following figure.

1. You can select Last 2 Hour(s),  Last 24 Hour(s), Last 3 Days),  Last 7 Day(s), and Last 15 Day(s) to view read details, as shown in the following figure.

2. The indicator Total Read (Row) is displayed as a column chart. You can click the chart to enlarge it, which displays the volume of incremental read messages of all tables in the task for each equal time segment, as shown in the following figure.

You can view the details of insert, update, and delete operations within a specified period, and switch between the current task and the source table.

3. The indicator Log Parsing Latency (added since V4.2.14.4) is displayed as an area line chart. If Current Task is selected on the Read and Write Statistics dialog, you can view the log parsing latency and total read volume. If you switch to the source table, only the total read volume is displayed.

The following chart shows the change trend of the log parsing latency of the capture task over the corresponding period.

Dependency

Note:

The Dependency tab has been added since V4.2.6.3.

Dependency displays the list of pipeline tasks and real-time tasks that depend on this real-time capture task, showing the name, type, status, and used tables of corresponding pipeline tasks and real-time tasks, as shown in the following figure.

You can click the task name to jump to the corresponding pipeline/real-time task.

Execution Log

Execution Logs displays the execution logs of this real-time capture task, as shown in the following figure.

You can filter the task execution logs, as shown in the following figure.

In the real-time capture task list, you can click All Tasks to view the execution status of all real-time capture tasks. 

Note:

This function has been supported since V4.2.6.3.

You can filter real-time capture tasks by connection name (supported since V4.2.14.1), task name, and latest startup time, as shown in the following figure.

You can click a task name to jump to the details page of that real-time capture task.