When integrating FineDataLink with standard platforms, you, the developer, often face issues such as inaccurate connections to cloud‑based application data sources, low efficiency, and high O&M costs. You may want a fast, accurate, and low-cost integration solution that simplifies the development workflow and improves connection efficiency.
This solution employs a cloud-based agent for forwarding and parsing, connecting to application data sources via scheduled pipeline tasks. It streamlines data acquisition, addresses the challenges of integrating multi-source heterogeneous data, and reduces time to value in data development. With the centralized task management and monitoring function, enterprises can quickly access and utilize data scattered across various systems/platforms, enabling data-driven business analysis and decision-making.
For data sources supported by scheduled pipeline tasks for reading and writing, see Data Sources Supported by FineDataLink.
No coding is required. You can configure scheduled pipeline tasks quickly by following a series of simple steps.
Automatic retry and resumption from checkpoints
FineDataLink provides a Retry After Failure function. After each successful execution of a scheduled pipeline task, the checkpoint position for incremental tables within the task will be updated. The next execution will start from the latest checkpoint, as shown in the following figure.
Automatic DDL change synchronization
It describes how to create a scheduled pipeline task and how to configure timed scheduling, result notifications, fault tolerance mechanisms, and log levels for scheduled pipeline tasks.
Scheduled Pipeline Task Configuration
Scheduled pipeline task example
It provides a simple example to help you understand the scheduled pipeline task function.
Application System (Cloud) Topic
For supported source types, if Synchronize Source Table Structure Change is enabled, DDL changes will be automatically synchronized during task execution, requiring no manual updates to field mappings.
Synchronizing DDL Changes Using a Scheduled Pipeline Task
Single scheduled pipeline task O&M
It describes how to move, rename, or delete pipeline tasks.
It describes how to edit, delete, or run scheduled pipeline tasks.
It describes the execution workflow and status of scheduled pipeline tasks.
Scheduled Pipeline Task Status and Table Status
It describes how to view the configuration details and execution records of scheduled pipeline tasks.
Scheduled Pipeline Task Configuration/Execution Detail Description
You can filter execution records by task status, task name, trigger method, presence of dirty data, presence of an exception table, or execution time to view task execution details.
You can view the configuration information of scheduled pipeline tasks, configure scheduling plans and task control settings, and edit or delete scheduled pipeline tasks.
It describes how administrators can assign the Management permission on Scheduled Pipeline to users, allowing them to view and edit scheduled pipeline tasks.
You (the super admin) can assign the Authorization permission on scheduled pipeline tasks to a user to make the user a sub-admin.
Then the sub-admin can assign the Management permission on Scheduled Pipeline to subordinate members so that they can view and edit the scheduled pipeline tasks.
You can filter operation logs of scheduled pipeline tasks.
You can adjust memory allocation and concurrency control for the Scheduled Pipeline module.
During field mapping, system fields will be added to the list. For details, see the following table.
The business primary key of the primary table, for example, order_id in an order table
The connection ID for a specific scheduled pipeline task
The timestamp when the data was written to a cloud MongoDB database (or the time when the data was fetched)
__dm_hash_id
The system-generated primary key for the primary table (It is a globally unique identifier.)
The __dm_hash_id value in the primary table corresponds to this row in the child table, which is present only in child tables and can be used to join child tables with their primary table
The auto-incrementing ID from the cloud MongoDB database
The ID of an individual task during synchronization (There may be multiple tasks for the same table within a single synchronization operation.)
滑鼠選中內容，快速回饋問題
滑鼠選中存在疑惑的內容，即可快速回饋問題，我們將會跟進處理。
不再提示
10s後關閉
Submitted successfully
Network busy