Real-Time Capture Task O&M

Version

Historical Version Updates

FineDataLink Version Functional Change 4.2.6.3 1. Added a Dependency tab to display the real-time pipeline tasks and real-time tasks that depend on the real-time capture task. 2. Displayed Log Parsing Latency when the source database was Oracle, MySQL, or SQL Server. 3. Added Real-Time Statistics and Historical Statistics tabs to the Tables Being Parsed tab page. 4. Allowed you to view the execution status of all data capture tasks. 4.2.10.4 Displayed the used and available storage of the Topic and its earliest data timestamp when you hovered over the icon beside the source table name. 4.2.13.1 Added periodic checks for the following items when SQL Server was used as the source database for real-time pipeline tasks and real-time tasks: The status of Agent and Capture jobs, with prompts in Execution Log The structures of the source tables currently being captured and their corresponding CDC tables, excluding field types. If the column count or field names did not match, the capture task generated task-level WARN logs. 4.2.14.1 1. Displayed Data Connection URL on the details page of the real-time capture task. 2. Displayed the Delete button in the upper-right corner of the real-time capture task. 3. Allowed you to manually pause real-time capture tasks. 4. Optimized the status of real-time capture tasks. 5. Allowed you to remove tables from one or more real-time capture tasks. 6. Allowed you to view backfill records and cancel backfill operations.

Function Description

After a real-time capture task is automatically created, you can view its execution details under System Management > Data Connection > Real-Time Capture Task, as shown in the following figure.

Filtering Real-Time Capture Tasks

You can filter real-time capture tasks by Connection Name, Task Name, Table Being Parsed, and task status, as shown in the following figure.

The following table describes the task statuses.

Renaming, Deleting, and Pausing Real-Time Capture Tasks

Renaming

You can rename real-time capture tasks, as shown in the following figure.

Deleting

You can delete real-time capture tasks in the Stopped or Runtime Error status, as shown in the following figure.

You can delete a capture task in the following scenarios:

• Existing capture tasks are no longer needed.

• Checkpoints of existing capture tasks are not required.

• Logs for capture task checkpoints are lost.

You can reset the capture task for a data connection by deleting the capture task.

Deletion logic:

Deleting a capture task removes all related information, including capture task checkpoints and logs, all Kafka Topics in this capture task, and all data within those Kafka Topics (including old Topics generated from backfill operations).

Pausing

This operation has been available since V4.2.14.1.

Real-time capture tasks in the Running status can be paused, as shown in the following figure.

You can pause a capture task in the following scenario:

If the real-time capture task places a heavy load on the database, click Pause to stop it immediately.

Pause logic:

Exceptions in real-time pipeline tasks or real-time tasks caused by manually pausing a capture task do not trigger the retry mechanism for those tasks.

• When you click Pause, the capture task enters the stopping process, stops capturing data from all tables in this task, switches its status from Running to Stopping, and pauses all backfill threads.

• Running real-time tasks that use this capture task abort with an error indicating that the capture task has been manually paused and synchronization of the corresponding tables has been aborted.

• Running real-time pipeline tasks that use this capture task abort with an error indicating that the capture task has been manually paused and real-time synchronization has been aborted.

Backfilling

Backfill Logic

1. Backfill Trigger Scenarios

2. Backfill Prerequisites

Backfill re-parses historical data from source database logs. Therefore, the prerequisite for successful backfill is that the transaction logs for the required period are retained in the source database.

If the logs have been cleared, backfill cannot proceed. You must instead perform a full and incremental synchronization of the pipeline.

Note:

PostgreSQL does not support backfill. Data capture can only start from the current time. If the required messages are no longer available in Kafka, an error will be reported.

3. Backfill Execution Logic

During backfill, the capture task starts a dedicated sub-collector to parse historical logs.

Note that:

• The main collector continues parsing the latest logs and is unaffected by backfill. Incremental synchronization through pipeline tasks continues without interruption.

• The sub-collector runs independently and stops automatically after finishing parsing the historical logs.

• The Kafka Topic used to store backfill data is temporary and is deleted after backfill completes and the data is merged.

Note:

If an error occurs in any table, the entire backfill process is aborted.

From FineDataLink V4.2.19.2, multiple independent backfill requests (involving more than one table) can be merged into fewer batches under specific conditions:

• When adding a table triggers backfill, the request is added to an existing queued batch with the same capture scope whenever possible, rather than placed in a separate batch for that table.

• For whole-database capture, backfill requests for different tables are added to the same batch whenever possible, as long as the batch is still queued. The earliest requested start time becomes the synchronization start time for the batch.

• For table-level capture, backfill requests for different tables can be placed in the same queued batch, but synchronization for each table starts from its own table-level checkpoint.

• If multiple tasks trigger backfill for the same table, the existing record for that table is reused, and references to all originating tasks are retained.

• If the same table already has a queued backfill request with a different column snapshot, the new request is not merged into the existing batch. Instead, it is placed in a new batch and executed in sequence.

Function Overview

1. When a table is added or a historical table resumes data capture, it appears in the real-time capture task with the Parsing status, or with the Queuing status if it is waiting in the queue.

2. From FineDataLink V4.2.14.1, you can view records of completed or ongoing backfills. On the real-time capture task details page, click View Backfill Record, as shown in the following figure.

The following figure shows the View Backfill Record page.

3. From FineDataLink V4.2.14.1, you can cancel an accidentally started backfill task that takes a long time.

A Cancel button is displayed when the backfill task is in the Queuing or Backfilling status.

After the backfill task is canceled:

• The Kafka Topic associated with the sub-collector and all data in the Topic are deleted.

• Running real-time tasks that depend on the backfill task abort with an error indicating that backfill for the table in the capture task has been manually canceled and synchronization has stopped.

• Running real-time pipeline tasks that depend on the backfill task abort with an error indicating that backfill for the table in the capture task has been manually canceled and synchronization has stopped. Synchronization of other tables whose backfill has not been canceled continues normally.

Select a real-time capture task to view its basic information, tables being parsed, dependencies, and execution logs, as shown in the following figure.

Basic Information

You can view the data connection name, read method, creation time, timestamp of the latest message read, log parsing latency (available since V4.2.6.3), and data connection URL (available since V4.2.14.1) of a real-time capture task, as shown in the following figure.

Log Parsing Latency is displayed only when the source database is Oracle, MySQL, or SQL Server.

Tables Being Parsed

Note:

The Real-Time Statistics and Historical Statistics tabs have been available since V4.2.6.3.

Real-Time Statistics

This tab displays all tables currently being captured by the capture task, as shown in the following figure.

1. The following table explains the table columns and the Total Read indicator.

2. From FineDataLink V4.2.14.1, you can delete tables from a real-time capture task in batches, as shown in the following figure.

3. From FineDataLink V4.2.14.1, prompts are displayed for unused tables in capture tasks, as shown in the following figure.

Historical Statistics

The tab is shown in the following figure.

1. You can select Last 2 Hour(s), Last 24 Hour(s), Last 3 Day(s), Last 7 Day(s), or Last 15 Day(s) to view read details, as shown in the following figure.

2. The Total Read (Row) indicator is displayed as a column chart. Click the chart to enlarge it and view the number of incremental messages read from all tables in the task during each equal time interval, as shown in the following figure.

You can view the details of insert, update, and delete operations within a specified period, and switch between the current task and source table.

3. The Log Parsing Latency indicator, available since V4.2.14.4, is displayed as an area chart. If Current Task is selected in the Read and Write Statistics dialog box, you can view the log parsing latency and total read volume. If you switch to Source Table, only the total read volume is displayed.

The following chart shows the trend of the capture task's log parsing latency over the selected period.

Dependency

Note:

The Dependency tab has been available since V4.2.6.3.

The Dependency tab lists the real-time pipeline tasks and real-time tasks that depend on this real-time capture task. It displays each task's name, type, status, and tables used, as shown in the following figure.

Click a task name to open the corresponding real-time pipeline task or real-time task.

Execution Logs

The Execution Logs tab displays the execution logs of the real-time capture task, as shown in the following figure.

You can filter the task execution logs, as shown in the following figure.

In the real-time capture task list, click All Tasks to view the execution status of all real-time capture tasks.

Note:

This function has been available since V4.2.6.3.

You can filter real-time capture tasks by Connection Name (available since V4.2.14.1), Task Name, and Latest Startup, as shown in the following figure.

Click a task name to open the details page of the corresponding real-time capture task.