Real-Time Task of the Data Development module is currently in the internal testing stage. To trial or obtain detailed information, you can send an email to international@fanruan.com or click at https://help.fanruan.com/finedatalink-en/.
FineDataLink Version
Functional Change
4.1.11.4
4.1.13.2
Added the IBM MQ Input operator and the MQTT Input operator.
When building a data warehouse, enterprises need to connect to real-time data sources such as Kafka, and synchronize the data read from Kafka to the data warehouse in real-time.
During the real-time data warehouse construction process, the intermediate data results need to be output to Kafka to support the calculation of the next layer.
Most of the data is in semi-structured JSON format, which needs to be parsed and converted into structured data before being stored in the database. Alternatively, the structured data needs to be serialized into JSON format before being placed into Kafka.
Data can be delivered from point A to point B in real-time via Real-Time Task. Data cleaning operations may be added during the delivery process, such as data parsing in the real-time data warehouse. The final results can be used in subsequent businesses to improve data utilization and timeliness, as well as meet business needs.
The Real-Time Task module of FineDataLink provides various types of nodes that can be used in combination to meet your diverse data processing needs.
A node is a basic unit of a real-time task. Multiple nodes form an execution process after being connected by lines and further form a complete real-time task. The basic information of currently available nodes is shown in the following table.
Type
Operator
Description
Data Input
Kafka Input
Allows reading data in real-time
CDC Input
Pulsar Input
MQTT Input
IBM MQ Input
Data Output
DB Table Output (Real-Time Task)
Allows outputting data to a specified database in real-time
Kafka Output
Allows outputting data to Kafka synchronously in real-time
Transformation
JSON Parsing
Parses JSON data and outputs data in the format of rows and columns
XML Parsing
Parses XML data into row-column format data
Field Setting
Selects and renames fields, and changes data types
New Calculation Column
Obtains a new field by referencing or calculating the original fields without affecting them
Data Filtering
Filters eligible data records
Field-to-Column Splitting
Splits field values according to specific rules (delimiters or the number of characters), where the split values form multiple new columns
Field-to-Row Splitting
Splits field values according to specific rules (delimiters), where the split values form a new column
Other
Remark
Helps users add comments to tasks and nodes
Order
Document
1
Configure the data sources for data reading and writing when configuring a real-time task.
Data Source Creation and Management
2
Enable the logs to read the data source.
Overview of Database Environment Preparation
3
Create and develop a real-time task.
See the document of each node.
4
Start the task.
5
Manage scheduled tasks and resources.
Single Real-Time Task Management
滑鼠選中內容，快速回饋問題
滑鼠選中存在疑惑的內容，即可快速回饋問題，我們將會跟進處理。
不再提示
10s後關閉
Submitted successfully
Network busy