Overview
Version
FineDataLink Version | Functional Change |
---|---|
1.9 | / |
4.0.14 | Added the node usage explanation that would be displayed when you hover the cursor over a node in the node list, allowing you to know how to use it quickly. |
4.0.28 | Added the Python Script node, allowing you to directly call Python scripts. |
4.0.29 | Added the Bat Script node, allowing you to directly call Bat script files in remote Windows environments. |
4.1.5.5 | Added the File Transfer node, allowing you to download/upload files between APIs, local storage, and SFTP/FTP servers, thus realizing file movement. |
4.1.6.3 | A scheduled task cannot be edited by multiple users simultaneously. |
4.1.9.1 | Supported parameter autocompletion in the SQL statement input box under Data Synchronization > DB Table Input, Body under Data Synchronization > API, SQL Script, the SQL statement input box under Parameter Assignment > DB Table Input, the SQL statement input box in DB Table Input, Body in API Input, Spark SQL, and Python. |
4.1.9.3 | You can use Recycle Bin to restore deleted tasks. You can use the Kettle Call node. |
Function Description
The Data Development module supports data extraction, transformation, and loading through the use of various nodes and operators on a visual page. It enables the automatic running of scheduled tasks through the Timed Scheduling function, helping you effortlessly build offline data warehouses and ensuring efficient and stable data production.

For details about some concepts of Data Development, see the section "Data Development" of FineDataLink Glossary.
Function Overview
Type | Description |
---|---|
Data Source | FineDataLink's Data Development module supports the processing and integration of data from various data sources. For details, see the section "Types of Data Sources Supported by Data Development" of Data Sources Supported by FineDataLink. |
Task Management | You can store the designed business processes under Data Development > Scheduled Task. You can manage scheduled tasks using Folder. |
Task Development | · Rich in capabilities You can use general synchronization nodes for cross-database data synchronization and data cleansing calculations. You can use various script nodes to call Python scripts, Shell scripts, and so on. You can use process nodes with general nodes for complex logic processing, such as conditional branch, loop execution, output result transfer, and other complex processes. · Easy to use You can quickly arrange a task by dragging and dropping nodes. An intelligent SQL statement editor is provided. For details about the node types supported by FineDataLink, see the section "Node Introduction." |
Task Schedule | · Task Control You can enable Timeout Limit and Retry After Failure for tasks. You can configure Task Priority, Dirty Data Tolerance, Result Notification, and Log Level Setting. · Scheduling Plan You can set Scheduling Start Time and Execution Frequency. You can set scheduling dependencies for tasks, for example, executing Task C only after tasks A and B have been executed successfully. For details about task scheduling, see Overview of Task Control. |
Task O&M | O&M personnel can flexibly schedule tasks and monitor the real-time running status, reducing the heavy workload. For details, see Scheduled Task O&M. |
Other | · Open capability: Rich open APIs are provided for you to implement functions such as triggering scheduled tasks by external systems. For details, see Description of Scheduled Task-Related API Calling. · Permission control: You can control permissions on page-based functions and data access. For details, see Scheduled Task Management Permission. · A scheduled task cannot be edited by multiple users simultaneously in FineDataLink 4.1.6.3 and later versions. · You can restore and manage deleted tasks in Recycle Bin in FineDataLink 4.1.9.3 and later versions. |
Node Introduction
FineDataLink's Data Development module provides various types of nodes that can be used in combination to meet your diverse data processing needs.
A node is a basic unit of a scheduled task. Multiple nodes form an execution process after being connected by lines and further form a complete scheduled task. FineDataLink provides the following nodes:
Type | Description | Node Name | Application Scenario |
---|---|---|---|
General | It is used for data synchronization. | Used to extract data from one database to another. | |
It is used for complex data cleansing calculations. | Used to perform operations such as data association and synchronous data deletion. | ||
It provides multiple ways to download/upload files. | File Transfer | Used to download/upload files from/to APIs, local servers, and SFTP/FTP servers, thus realizing file movement. Note: This node is used to synchronize files, not extracting the data in the files. | |
For details about differences between Data Synchronization and Data Transformation, see Differences Between Data Synchronization and Data Transformation. | |||
Script | It is used to process data by controlling the script syntax. | Used to write SQL statements to process the data in the database. | |
Shell Script | Used to connect to external independent data processing processes such as Kettle tasks and Python calculation tasks by calling shell scripts. | ||
Python Script | Used to call Python scripts. | ||
Bat Script | Used to call batch files in remote Windows environments. | ||
Process | It is used for complex logic processing. | Used to output the upstream node results as parameters for use by downstream nodes. | |
Conditional Branch | Used to determine whether to run the downstream node or which of the downstream nodes to run based on a condition from the upstream node or the system. | ||
Virtual Node | Used to achieve the parallel running of multiple upstream nodes. | ||
Invocation Task | Used to call other tasks in the current task and set the dependency between tasks to realize task orchestration. | ||
Notification | Used to notify specified users of task scheduling results via WeCom group robot messages/email/DingTalk messages/SMS. | ||
Loop Container | Used for define loops for fetching data. | ||
Other | It is used to add remarks for tasks. | Used to add remarks to a node or task, such as the task application scenario. |
The node usage explanation appears when you hover the cursor over a node in the node list, helping you quickly know how to use the node. See the following figure:
Processes
Order | Description | Document |
---|---|---|
1 | Configure the data source for data read and data write when designing a scheduled task. | |
2 | Create and develop a scheduled task. | See the document of each node. |
3 | Debug the task according to the task running log after manually executing the task. | |
4 | Publish the task. | / |
5 | Set a scheduling plan. | |
6 | Manage scheduled tasks and resources. |
Notes
SQL Statement Comment Description
In versions before 4.0.17, the processing of SQL statement comments is ignored. In FineDataLink 4.0.17 and later versions, the SQL statement comments are created in the database.
Ensure the format of SQL statement comments is correct: A space is required after a single-line comment. The Hive data source does not support multi-line comments.