4.2.7 Update Log

Supporting Post-distribution Data Processing in Data Transformation

Before optimization:

• Data Distribution was available only to operators immediately preceding output operators. To process data from the same source differently and output results separately, you had to create two transformation jobs with partially repeated logic via intermediate tables, which was cumbersome.

• Detailed output information of each data flow was displayed in Statistics, but the execution result of each data flow was not displayed in Log.

After optimization:

1. You can process and output data after distribution within a single Data Transformation node, as shown in the following figure.

2. You can view the execution log and statistics of the Data Transformation node on the Data Transformation page, as shown in the following figure.

Turning the DB Table Output (Transaction) Operator into a Standard Function

The DB Table Output (Transaction) operator has been converted to the Transaction Control function that lies in Write Method in the DB Table Output operator and the Data Synchronization node, as shown in the following figure.

Optimizing Data Writing Logic

To minimize transactional downtime of the target database and prevent write failures after table truncation, a Table Emptying Method setting item and corresponding options of Strategy for Primary Key Conflict are added since this version, as shown in the following figure.

Supporting SelectDB V3.0

FineDataLink supports connection to the SelectDB V3.0 database.

Expanding Supported Data Sources and Optimizing User Experience for Lineage Analysis

1. When you read data from the following data sources and set Configuration Method in Data Source to SQL, the Lineage Analysis function applies: Db2, SQLite, HSQLDB, Informix, DM, KingBase, TiDB, ClickHouse, PolarDB PostgreSQL, Presto, GaussDB 200, GBase8a, and Alibaba Cloud's MaxCompute (formerly referred to as ODPS).

2. The pipeline task lineage has been optimized.

Before optimization: When you viewed the lineage of a source table (for example, Table A) in a pipeline task, all target tables in the task would be displayed.

After optimization: Only a one-to-one table relationship will be displayed, regardless of whether Extended Lineage is enabled. In the above example, only the target table of Table A will be displayed.

3. The parsing of INSERT syntax has been optimized.

Before optimization: The source and target ends in INSERT statements were parsed as reference relationships.

After optimization: The source and target ends can be parsed correctly.

Supporting Partition Tables in Doris and StarRocks Databases

1. Scheduled Task and Data Service support data reading from partition tables in Doris and StarRocks databases.

2. Scheduled Task and Pipeline Task support data writing into partition tables in Doris and StarRocks databases, as shown in the following figure.

3. You can specify the partition key when setting the target table of scheduled tasks or pipeline tasks to Auto Created Table, as shown in the following figure.

Supporting GaussDB 100 and GaussDB T Data Sources

FineDataLink supports connection to GaussDB 100 and GaussDB T data sources with the following functions available:

• Data reading/writing using scheduled tasks

• Data writing using real-time tasks

• Data writing using pipeline tasks

• The Data Service function

• The Database Table Management function

Supporting Lark's Message Card

You can set the message content in a card format when setting Notification Channel to Lark App Notification or Lark Chatbot.

Additionally, the display effect of Markdown-formatted messages has been optimized to improve message readability.