Lineage Analysis- FineDataLink Help Document

Last update: November 04, 2024

Overview

Version

FineDataLink Version	Functional Change
4.1.11.1	/
4.1.11.4	Supported Greenplum data sources if SQL is selected as the configuration method.

Application Scenario

You may want to find corresponding tasks as well as upstream and downstream tables based on data tables. For example, the IT personnel who are currently responsible for development need to view the upstream and downstream relationships between database tables clearly and visually.

Function Description

You can view the lineage relationships between data tables, APIs (data service), nodes of scheduled tasks, and pipeline tasks by clicking database/data table names in the Data Management module.

Prerequisite

The Data Lineage function point has been registered.

Upgrade Compatibility

If lineage is not displayed after the update, the following are possible reasons:

The data connection used in the task failed during the update.
The version of the database driver is too low.

Configuration Supported by Lineage Analysis

Configuration Item	Supported Data Source	Supported Configuration
Select Table Selection as Configuration Method on the source end	All data sources are supported.	Select a table to fetch data in the SQL Script, (the DB Table Input operator of) Data Synchronization, and Parameter Assignment nodes of a scheduled task. Select a table as the data source of a pipeline task. Select a table to fetch data for an API task of the Data Service module.
Select a table/database as the data destination.	All data sources are supported.	The DB Table Output operator of Data Transformation and Data Synchronization nodes of a scheduled task Select a target table for the data of a pipeline task.
Select SQL as Configuration Method on the source end	Only MySQL, Oracle, SQL Server, PostgreSQL, and Greenplum data sources are supported.	Use SQL statements to fetch data in the SQL Script, (the DB Table Input operator of) Data Synchronization, and Parameter Assignment nodes of a scheduled task. Use SQL statements to fetch data for an API task of the Data Service module.

Configuration Item

Supported Data Source

Supported Configuration

Select Table Selection as Configuration Method on the source end

All data sources are supported.

Select a table to fetch data in the SQL Script, (the DB Table Input operator of) Data Synchronization, and Parameter Assignment nodes of a scheduled task.
Select a table as the data source of a pipeline task.
Select a table to fetch data for an API task of the Data Service module.

Select a table/database as the data destination.

All data sources are supported.

The DB Table Output operator of Data Transformation and Data Synchronization nodes of a scheduled task
Select a target table for the data of a pipeline task.

Select SQL as Configuration Method on the source end

Only MySQL, Oracle, SQL Server, PostgreSQL, and Greenplum data sources are supported.

Use SQL statements to fetch data in the SQL Script, (the DB Table Input operator of) Data Synchronization, and Parameter Assignment nodes of a scheduled task.
Use SQL statements to fetch data for an API task of the Data Service module.

Table Lineage in the Scheduled Task

For example, in the Data Synchronization node, you use the following SQL statement to join two data tables, and the task is published to Production Mode.

Find a data table DEMO_PRODUCT of the scheduled task in the Data Management module, click Lineage Analysis, and enable Paralogue to view the lineage relationship of the data table SQL statements used by the scheduled task, as shown in the following figure.

Note:

All related objects along the entire chain will be displayed.

You can also view detailed information of the data table, as shown in the following figure.

Click Data Synchronization or other nodes to view the node detail in the pop-up box, where you can click Task Running Record to view the task running record and the button to open the task, as shown in the following figure.

Table Lineage in the Pipeline Task

Create a pipeline task with the AREA_INFO data table, as shown in the following figure.

Go to Data Management, find the AREA_INFO data table of the pipeline task, and click Lineage Analysis to view the lineage of the data table, as shown in the following figure.

You can also view detailed information of the data table, as shown in the following figure.

Click the pipeline task to view details or make yourself redirected to the task page, as shown in the following figure.

Table Lineage in the API Task

Create an API task with the AREA_INFO data table and take the task online, as shown in the following figure.

Go to the Data Management module, find the AREA_INFO data table of the API task, and click Lineage Analysis to view the lineage of the data table, as shown in the following figure.

Click the API task to view API running records or make yourself redirected to the task page, as shown in the following figure.

Notes

Function Usage

The preceding SQL and subsequent SQL in the DB Output (Transaction) operator, including the temporary tables generated by transaction rollbacks, are not currently parsed in the lineage.
Table names are displayed as original names instead of the alias names after as in SQL statements.
For partitioned tables, only the main table is displayed. The partition table information will not be displayed.
View: Lineage relationship can be parsed if you select Table Selection or SQL as Configuration Method.
Stored procedures do not allow parsing table relationships.

Lineage Detection Failure

Problem:

In a scheduled task, data is synchronized to table a of MySQL (the task has been executed). When you select table a in Data Management and click Lineage Analysis, the content is blank.

Cause:

An issue with the business database during the update occurs. Republish the scheduled task.