FineDataLink Glossary

Data Development

Develop and arrange tasks through SQL statements or visual operations. 

Synchronize data in real time with high performance in the case of large data volume or standard table structure.

Encapsulate the standardized data after processing as standard APIs and release them for external systems to call.

It refers to the data flow between the input widgets and the output widgets, focusing on the processing of each row of records and each column of data. There are various operators for completing data input, output, transformation, and other operations.

A data flow (a data transformation node) only provides the following three types of operators, and does not contain combinatorial and process-type operators:

• An example of output operators: DB Table Output.   

• An example of processing operators: Data Association.

• An example of input operators: DB Table Input.

Obtain data from specified Jiandaoyun forms.

Delete the data rows that exist in the target table but have been deleted in the input source by comparing field values. It includes:

• Physical Deletion: It deletes data.

• Logical Deletion: It does not delete data, and only adds deletion identifiers.

3.2

Note: This operator has been deleted since Version 4.0.18.

Advanced Data Source Control - NoSQL

Join

Data Association

Advanced

Support cross-database and cross-source connections.

The join methods include:

• Left Join 

• Right Join 

• Inner Join

• Full Outer Join

These join methods are consistent with how database tables are joined. You can get the join results by defining the association fields and conditions. It requires more than two inputs and only one output.

Data Comparison

• Select the original table and the target table.

• Configure the logical primary key.   

• Configure the comparison field.   

• Set the identifier.

Union All

Merge multiple tables by rows, and output a combined table.

Transformation

Field Setting

For adjusting the field name and type.

• Set Column: Select and delete fields.

• Modify Column: Modify the field name and type.

Column to Row

For changing the row and column structure of the data table to meet the needs of conversion between one-dimensional tables and two-dimensional tables.

Convert columns in the input data table to rows.

• Column to Row (also known as unpivoting): Convert one-row multi-column data into multi-row one-column data. Usually, the converted column is named by the value of a cell to identify the original data.

Row to Column

For converting the rows in the data table to columns.

Convert rows in the input data table to columns.

• Row to Column (also known as pivoting): Convert multi-row one-column data into one-row multi-column data. Usually, the converted column is named by the classified value of a column, and multiple rows of data corresponding to this value are displayed on a row.

JSON Parsing

For parsing JSON data and output data in the format of rows and columns.

Obtain JSON data output by the upstream node, parse them into row-and-column format data, and output parsed data to the downstream node.

XML Parsing

For specifying the parsing strategy and parsing the input XML data into row-and-column format data.

Specify the parsing strategy and parse the input XML data into row-and-column format data.

JSON Generation

For selecting fields to generate JSON objects.

Select fields and convert table data into multiple JSON objects, which can be nested.

New Calculation Column

For generating new columns through calculation.

Support formula calculation or logical mapping of constants, parameters, and other fields, and place the results into a new column for subsequent operations or output.

Data Filtering

For filtering eligible data records.

Filter eligible data records.

Group Summary

It refers to merging the same data into one group based on conditions and summarizing and calculating data based on the grouped data.

For splitting field values according to specific rules (delimiters), where the split values form a new column.

Field-to-Column Splitting

Laboratory

Spark SQL

For improving scenario coverage by providing flexible structured data processing.

With the built-in Spark calculation engine and Spark SQL operators, you can obtain the data output by the upstream node, query and process them using Spark SQL, and output them to the downstream node.

3.6

It can be used as an input operator in V4.0.17.

For running Python scripts to process complex data.

A step flow, also called workflow, is the arrangement of steps. Each step is relatively independent and runs in the specified order without the flow of data rows.

Each step is a closed loop from input to output.

General

Data Synchronization

For completing data synchronization (between input and output) quickly. Data transformation is not supported during data synchronization.

Support multiple methods to fetch data, such as fetching data by API, SQL statement, and file. Memory calculation is not required as there is no data processing during the process.

It applies to the following scenarios where:

• Rapid synchronization of data tables is needed.

• The calculation is finished during data retrieval, and no calculation or transformation is needed in the process.

• The target database has strong computing ability or the data volume is large. Synchronize data to the target database and then use SQL statements for further development.

Data Transformation

Advanced

For transforming and processing data between the input and output nodes.

Note: The Data Transformation node can be a node in a step flow.

Complex data processing such as data association, transformation, and cleaning between table input and output nodes is supported based on data synchronization.

Data Transformation is a data flow in essence but relies on the memory computing engine since it involves data processing. It is suitable for data development with small data volumes (ten million records and below), and the computing performance is related to memory configuration.

Script

SQL Script

For issuing SQL statements to the specified relational database and execution.

Write SQL statements to perform operations on tables and data such as creation, update, deletion, read, association, and summary.

Shell Script

For running the executable Shell script on a remote machine through the remote SSH connection.

Run the executable Shell script on a remote machine through a configured SSH connection.

Python Script

For running Python scripts as an extension of data development tasks. For example, you can use Python programs to independently process data in files that cannot be read currently.

Bat Script

For running the executable Batch script on a remote machine through the remote connection.

Process

Conditional Branch

For conducting conditional judgment in a step flow. 

Task Call

For calling other tasks to complete the cross-task arrangement.

Virtual Node

Null operation.

A virtual node is a null operation and functions as a middle point to connect multiple upstream nodes and multiple downstream nodes. It can be used for process design.

Loop Container

For the loop execution of multiple nodes.

Parameter Assignment

Output the read data as the parameter for use by downstream nodes.

Notification

Notification

Notifications can be sent through email, SMS, platform, enterprise WeChat (group robot notification and App notification), and DingTalk.

You can customize the notification content.

V3.6

V4.0.1

V4.0.3

Connection

Execution Judgment on Connection

Right-click a connection line in a step flow and configure the execution condition, with options including Execute Unconditionally, Execute on Success, and Execute on Failure. 

Right-click a node in a process flow, and click Execution Judgment. You can customize the method for multiple execution conditions to take effect (All and Anyone) to control the dependencies of nodes in the task flexibly.

Others

Remark

You can customize the content and format.