Note:Real-Time Task of the Data Development module is currently in the internal testing stage. To trial or obtain detailed information, you can send an email to international@fanruan.com or click 
at https://help.fanruan.com/finedatalink-en/.
Overview
Version
FineDataLink Version | Functional Change |
|---|---|
4.1.11.4 | / |
4.1.13.2 | Added the IBM MQ Input operator and the MQTT Input operator. |
Application Scenario
When building a data warehouse, enterprises need to connect to real-time data sources such as Kafka, and synchronize the data read from Kafka to the data warehouse in real-time.
During the real-time data warehouse construction process, the intermediate data results need to be output to Kafka to support the calculation of the next layer.
Most of the data is in semi-structured JSON format, which needs to be parsed and converted into structured data before being stored in the database. Alternatively, the structured data needs to be serialized into JSON format before being placed into Kafka.
Function Description
Data can be delivered from point A to point B in real-time via Real-Time Task. Data cleaning operations may be added during the delivery process, such as data parsing in the real-time data warehouse. The final results can be used in subsequent businesses to improve data utilization and timeliness, as well as meet business needs.
Node Description
The Real-Time Task module of FineDataLink provides various types of nodes that can be used in combination to meet your diverse data processing needs.
A node is a basic unit of a real-time task. Multiple nodes form an execution process after being connected by lines and further form a complete real-time task. The basic information of currently available nodes is shown in the following table.
Type | Operator | Description |
|---|---|---|
Data Input | Kafka Input | Allows reading data in real-time |
CDC Input | Allows reading data in real-time | |
Pulsar Input | Allows reading data in real-time | |
MQTT Input | Allows reading data in real-time | |
IBM MQ Input | Allows reading data in real-time | |
Data Output | Allows outputting data to a specified database in real-time | |
Kafka Output | Allows outputting data to Kafka synchronously in real-time | |
Transformation | JSON Parsing | Parses JSON data and outputs data in the format of rows and columns |
Parses XML data into row-column format data | ||
Field Setting | Selects and renames fields, and changes data types | |
New Calculation Column | Obtains a new field by referencing or calculating the original fields without affecting them | |
Data Filtering | Filters eligible data records | |
Field-to-Column Splitting | Splits field values according to specific rules (delimiters or the number of characters), where the split values form multiple new columns | |
Field-to-Row Splitting | Splits field values according to specific rules (delimiters), where the split values form a new column | |
Other | Remark | Helps users add comments to tasks and nodes |
Procedure
Order | Description | Document |
|---|---|---|
1 | Configure the data sources for data reading and writing when configuring a real-time task. | |
2 | Enable the logs to read the data source. | |
3 | Create and develop a real-time task. | See the document of each node. |
4 | Start the task. | / |
5 | Manage scheduled tasks and resources. | Single Real-Time Task Management |
Previous：Overview of Scheduled Task
