Note:
You may need to read data from databases in real time and output it to specified databases in real time after processing for reporting and real-time dashboard production and display.
The CDC Input operator in FineDataLink's Real-Time Task enables real-time data reading.
Supported data sources include Db2, MySQL, Oracle, PostgreSQL, and SQL Server. Enable database logging before using these data sources as input sources. For details, see Overview of Database Environment Preparation.
Supported read modes are database-dependent.
For Oracle databases, the read mode is LogMiner.
You can configure initial offsets for tasks to meet different scenario requirements.
Full + Incremental Synchronization: When the task runs for the first time, all existing data in the database table will be synchronized and incremental changes will be continuously synchronized from the database. When the task restarts, the synchronization will resume from the last recorded checkpoint. If the checkpoint status is lost, the synchronization logic used during the first task start will work.
Incremental Synchronization Only: You can set Incremental Sync Start Point to Task Startup Time or Custom Time.
1. If you select Custom Time, you need to select the date and time using the date-time picker.
2. Task Startup Time: When the task runs for the first time, incremental changes from the task startup time will be synchronized. When the task restarts, the synchronization will resume from the last recorded checkpoint.
3. Custom Time: When the task runs for the first time, the incremental synchronization starts from the specified start point. When the task restarts, the synchronization will resume from the last recorded checkpoint.
4. If the checkpoint status is lost, the synchronization logic used during the first task start will work.
Take processing data fetched from a MySQL database in real time and outputting results to an Oracle database in real time as an example.
Log into FineDataLink, choose Data Development > Real-Time Task, and create a real-time task, as shown in the following figure.
Drag a CDC Input operator onto the task editing page, and set Data Source, Source Table, Read Mode, and Synchronization Type, as shown in the following figure.
Click Data Preview to view the fetched data, as shown in the following figure.
Process the fetched data. For example, use a Data Filtering operator to obtain data whose value of owner_region is North China or East China, as shown in the following figure.
The result data is shown in the following figure.
Use a DB Table Output operator to output data after real-time processing to an Oracle database, as shown in the following figure.
Click Start to start the real-time task, as shown in the following figure.
The result of the first synchronization is displayed in Statistics, as shown in the following figure.
If a data record is added to the source database, the input row count will increase by one, as shown in the following figure.
滑鼠選中內容，快速回饋問題
滑鼠選中存在疑惑的內容，即可快速回饋問題，我們將會跟進處理。
不再提示
10s後關閉
Submitted successfully
Network busy