Synchronizing DDL Changes Using Scheduled Pipeline

Version

Application Scenario

When a scheduled pipeline task synchronizes data from application data sources, the source table structure may undergo DDL changes due to business adjustments (such as adding/deleting tables/fields). If these changes are not detected and handled promptly, synchronization errors and data inconsistencies may occur, affecting downstream business operations.

You want to detect these changes quickly and have the flexibility to manually update mappings or automatically synchronize certain DDL changes.

Function Description

Source DDL change detection: If the source table or its structure changes, the task details page displays the corresponding change information.

Synchronize Source Table Structure Change: For supported source types, if Synchronize Source Table Structure Change is enabled, DDL changes are automatically synchronized during task execution, requiring no manual updates to field mappings.

The following table lists the detectable changes to source tables and source table structures and handling methods:

On the task details page, if the source table structure has changed, the prompt "The mapping of x table(s) can be updated." will be displayed. You can click Filter to list tables with updatable mappings, as shown in the following figure.

Click the  icon to enter the editing page, as shown in the following figure.

Updating Field Mappings

If the source table structure has changed, click Update Mapping to update the source table structure, as shown in the following figure.

The Update Mapping table displays the changes in source fields. Click Update Mapping and Target Table to enter the Update Target Table dialog. For details about updating the target table, see the "Updating the Target Table" section of this document.

After you click Update Mapping, the system first updates the field mappings of the source table and highlights the conflict content in the target table in red in Field Mapping. You need to click the Update Target Table button to adjust the target table. For details about updating the target table, see the "Updating the Target Table" section of this document.

Updating the Target Table

If the source field has changed, you can modify the field mapping by selecting the existing fields or manually adding fields. After you confirm the changes in the mapping relationships, the DDL statement for the target table is generated and executed.

• You can create, delete, and rename fields in the target table.

• You can modify the data type, length, and precision of fields in the target table.

• You can configure Not Null, Primary Key, and Remark for the target table.

• Change records are displayed. You can revert changes.

1. Click Update Target Table. The popup displays the actual fields of the target table in order. You can adjust the field structure, as shown in the following figure.

2. You can use the one-click apply function. The field name is used as the unique identifier (based on the Map Fields with Same Name logic) to compare the structures of the source and target tables. If any inconsistencies are detected, you can synchronize the source table structure changes to the target table with one click.

The following table describes the change records.

Change Record	Description
Added	Fields that exist in the source table but not in the target table are marked as Added.
Modified	Fields with the same name exist in the target table, but the Type, Remark, Primary Key, or Not Null properties differ from those in the source table. Note: Remark, Primary Key, and Not Null settings can be applied if they can be obtained from the source table. Existing values of the target table are preserved otherwise.
Deleted	Fields that exist in the target table but not in the source table are marked as Deleted.

Note:

If you click Apply and then make changes on this page that cause the source table and target table structures to become inconsistent, the Apply-related prompt will not appear again.

3. Update statements are generated based on the update content. Click Execute Update Statement, as shown in the following figure.

For certain source table structure changes, you can enable Synchronize Source Table Structure Change, which ensures automatic change synchronization during task execution without the need for manual mapping updates, as shown in the following figure.

The following table describes the change synchronization effects in various scenarios with this function enabled.

After the task runs, you can view the execution logs in Execution Record to check source table structure changes, as shown in the following figure.