Overview of Real-Time Task

Application Scenario

1. You want to read data from source systems in real time and write real-time data to the target database. For example:

• New order and contract data needs to be synchronized from the MySQL database of a CRM business system to the data warehouse in real time.

• Data from each branch of a group-level enterprise needs to be written to Kafka in real time for the headquarters' data lake to consume.

2. You want to process the read data in real time and then write the result data to the data warehouse. For example:

• Water data of a water utility, such as upstream and downstream water levels and water levels of multiple tributaries, needs to be aggregated and calculated in real time before being presented.

• Manufacturing enterprises need to display metrics, such as production line yield rates, utilization rates, factory efficiency, losses, and working hours, in real time.

Function Description

You can use Real-Time Task to deliver data from Point A to Point B in real time. Data cleaning, such as real-time data parsing in the data warehouse, is supported during delivery. Downstream business processes can use the final results. Real-time delivery with in-flight cleansing reduces latency and converts raw data into consumable data, improving both timeliness and utilization.

1. Function points related to Real-Time Task must be registered. For details, see the "Function Point Limit" section of Registration Introduction.

2. To create a real-time task in Data Development, you must have Management permission on a specific folder or all real-time tasks. For details, see Scheduled Task Management Permission.

3. Cluster projects of V5.0.1.2 and later versions support real-time tasks.

4. From FineDataLink V5.0.1.5, if a WebSocket Input, MQTT Input, IBM MQ Input, or Pulsar Input operator is used in a real-time task, you cannot use Group Summary, Flink SQL, or Data Association in the task.

1. Data source:

For data sources supported by real-time tasks, see Real-Time Task.

2. Synchronization scenario:

In a real-time task, only single-table real-time synchronization is supported. The synchronization types are described below:

• Full + Incremental Synchronization: First synchronize all inventory data, and then continuously synchronize the changes.

• Incremental Synchronization Only: Continuously synchronize data from the specified start point.

3. Task development:

The Real-Time Task module of FineDataLink provides various types of operators that can be used in combination to meet your diverse data processing needs.

An operator is a basic unit of a real-time task. Multiple operators connected by lines define the execution flow and form a complete real-time task. The following table describes currently available operators:

4. Task configuration:

After developing a real-time task, you can configure Retry After Failure, Dirty Data Tolerance, Result Notification, Log Level Setting, and other settings for it. For details, see Single Real-Time Task Management.

5. Task O&M:

• You can view the execution status of real-time tasks and start or pause them in batches. For details, see Scheduled Task O&M – Task Management.

• Two startup modes are available for real-time tasks:

• Start: The task starts from the last checkpoint.

• Restart: You can click Restart to initialize the task, in which case the synchronization will restart based on the synchronization type set in the input operator.

• If the upstream input operators of Data Association include multiple real-time data sources, or if the Group Summary operator is used, the following applies after the real-time task starts:    

• If the task stops before the full synchronization phase completes (for example, due to dirty data), you must restart it to ensure data consistency.