Real-Time Pipeline Task Management

Version

Function Description

1. Folder-based pipeline task management is supported. You can copy, move, rename, export, and delete real-time pipeline tasks, as shown in the following figure.

2. This document describes how to edit real-time pipeline tasks.

You can view and edit all pipeline tasks within your permission scope in the task list.

Versions Before V4.1.11.1

When you click a folder in Data Pipeline, all pipeline tasks within this folder will be displayed on the right. You can adjust the display format of pipeline tasks (whether in the card view or the list view).

Note

:

Starting from V4.1.11.1, the display formats (card or list) of pipeline tasks have been removed.

General Function Description

1. You can manage pipeline tasks by folder.

2. You can drag a pipeline task to a specified folder, as shown in the following figure.

Click the  icon on the right side of a pipeline task, and you can rename, delete, or export the task, as well as move or copy the task to a specified path, as shown in the following figure.

Note

:

A real-time pipeline task can be deleted only when its status is Draft, To Be Started, Paused, or Aborted. For details, see Description of Pipeline Task Status and Table Status.

1. In FineDataLink of V4.1.6.5 and later versions, you can drag and drop pipeline tasks/folders to change their positions.

• All users can manually drag and drop pipeline tasks/folders for which they have permission.

• The new order works on all users' pages.

• For example, a user sees top-down pipeline tasks 5, 1, and 7, and an admin sees top-down pipeline tasks 5, 6, 1, 9, and 7. If the user places the pipeline task 7 between pipeline tasks 5 and 1, then the pipeline task 7 is shown between pipeline tasks 6 and 1 on the admin’s page, and the pipeline task 7 is placed near the lower target node. (In this scenario, pipeline tasks 5 and 1 are target nodes, and pipeline task 1 is the lower one.) The same rule applies to folders.

2. In FineDataLink of V4.1.6.5 and later versions, the Move to button is added.

If you click the Move to button on the right side of a pipeline task, the folder to which the task belongs will be changed, but the order of the task cannot be changed. For example, if you click the Move to button to move a pipeline task in Folder A to Folder B, the task will be located at the end of Folder B. If the task is moved back to Folder A, the task will be located at the end of Folder A. The Move to button is effective for all users.

Modifying Task Configuration

Note:

This section applies to FineDataLink of V4.2.11.3 and later versions.

After a real-time pipeline task has been running for a period, you may want to modify its configuration. FineDataLink provides an entry for modifying the specific configuration, as shown in the following figure.

The following table describes the editable setting items.

Setting Item	Granularity	Restriction	Editable Item Description
Advanced Setting	Task-level	Modification is permitted when the task status is To Be Started, Paused, or Aborted.	All setting items are editable.
Task Control
Table Adding	Task-level	Tables can be added when the task status is To Be Started, Starting, Running, Paused, or Aborted.	Note: You do not need to pause the real-time pipeline task. You can add a synchronization object and start it once the configuration is finished. The system allows adding synchronization objects: 1. Entry: 2. When adding a synchronization object, you can configure the synchronization type, field mapping, write method, and other settings. If you set Synchronization Type to Incremental Synchronization Only, you can specify the synchronization start point (available only for data sources that support synchronization from a specified start point). If you switch to Full + Incremental Synchronization, the incremental synchronization start point of the table is the moment the synchronization object is started.
Field Mapping and Write Method	Table-level	Table Level Modification is permitted when the task status is To Be Started, Paused, or Aborted.	You can delete synchronization objects. You can edit synchronization objects. When modifying a synchronization object, you can modify Synchronization Type, Field Mapping, Write Method, and other settings.   Batch operations are supported, including adding (Add Table), deleting (Batch Delete), and editing (Table Editing) synchronization objects. You can also click Set Synchronization Type to set the synchronization type for all tables. Note: For edits on the synchronization object:1. If you do not change the synchronization type or the start point of incremental synchronization, the synchronization of the object will start from the breakpoint the next time the object synchronization is executed. 2. If you change the synchronization type or the start point of incremental synchronization, the system will discard the breakpoint, read-write statistics, and dirty data of the synchronization object. It will restart synchronization based on the new synchronization type and the start point of incremental synchronization, which is equivalent to resetting the synchronization object.

Resetting Tasks

Application Scenario:

When source tables undergo changes or exceptions occur (such as log expiration and checkpoint failures), you can use this function to restore the pipeline to normal operation.

Function Description:

Starting from V4.2.11.3, the Reset Task button is available when a real-time pipeline task has been executed, and its status is Pausing/Paused/Aborted.

The functions of this button are as follows:

Pausing Table Synchronization

You can pause a table when its synchronization status is Starting, Pending Full Synchronization, Full Synchronization in Progress, Pending Incremental Synchronization, Incremental Synchronization in Progress, Dirty Data Processing in Progress, or Retrying.

 1. You can click the Pause Synchronization for All Tables button, or pause the synchronization of a single table, as shown in the following figure.

 2. You can pause multiple tables as needed, as shown in the following figure.

Starting Table Synchronization

Note:

You can start a table when its synchronization status is To Be Started, Paused, or Aborted.

1. You can click the Start Synchronization for All Tables button, or start the synchronization of a single table, as shown in the following figure.

2. You can start multiple tables as needed, as shown in the following figure.

Resynchronizing Tables

You can resynchronize a table when its synchronization status is Paused or Aborted.

Resynchronizing a Single Table:

You can resynchronize a single table by clicking the  icon, as shown in the following figure.

• Full resynchronization of single tables is supported. If you click Resync, the task will empty the target table and execute full synchronization again, and perform incremental synchronization after full synchronization is completed.

• If the task is enabled with full resynchronization, the log statistics will be reset (including input rows and output rows).

Note:

If Logical Deletion is enabled, the target table will be cleared for data rewriting during resynchronization. The rewriting uses the INSERT logic. A prompt will appear, indicating that logical deletion data generated during task execution will be cleared.

Batch resynchronization:

You can select multiple tables and click the Resync button to perform batch resynchronization, as shown in the following figure.