Overview of Data Transformation

  • Last update: December 10, 2024
  • Overview

    Version Description

    FineDataLink VersionFunctional Change
    4.1.2

    Added the Field-to-Column Splitting, Field-to-Row Splitting, and Group Summary operators.

    4.1.3

    When Data Source in the DB Table Input operator was set to DB Table Input:

    • With Configuration Method set to Table Selection, you could select a source table for data synchronization and enable Parallel Read

    • You could call stored procedures if the data source was a MySQL, Oracle, or SQL Server database.

    4.1.6.4

    Added the MongoDB Output operator, allowing you to output data to a MongoDB database.

    4.1.11.3

    Added the Elasticsearch Output operator, allowing you to output data to an Elasticsearch database.

    4.2.0.2

    Added the Elasticsearch Input operator, allowing you to fetch data from a specified Elasticsearch database.

    Application Scenario

    The Data Synchronization node supports cross-database data synchronization. However, if you want to synchronize data to a database after complex processing (such as JSON parsing and multi-table association), you need to use the Data Transformation function, as shown in the following figure.

    iconNote:
    If you want to synchronize a large amount of data to the target database without complex data processing, you can use the Data Synchronization node.

    2024-12-09_17-10-22.png

    Function Description

    Data Transformation provides various types of operators such as input, output, and transformation operators, which allow you to achieve complex data processing.

    iconNote:
    For details about the differences between Data Synchronization and Data Transformation, see Differences Between Data Synchronization and Data Transformation.

    2024-12-10_11-28-50.png

    Function List

    Enter the Data Synchronization node. The following figure shows the Data Synchronization page.

    2024-12-09_17-11-38.png

    The following table describes the operators in the Data Synchronization node.

    TypeOperatorDescription

    Data Input

    DB Table Input

    Allows you to fetch data from tables in relational databases.

    For details, see Data Sources Supported by FineDataLink.

    In V4.1.3 and later versions, when DB Table Input is selected as the data source type:

    • With Configuration Method set to Table Selection, you can select a source table for data synchronization and enable Parallel Read

    • You can call stored procedures if the data source is a MySQL, Oracle, or SQL Server database.

    API Input

    Allows you to fetch data from APIs (including RESTful APIs and WebService APIs).

    File Input

    Allows you to fetch Excel, CSV, and TXT file data from local FineDataLink servers and FTP/SFTP servers.

    Jodoo Input

    Allows you to fetch data from Jodoo forms.

    MongoDB Input

    Allows you to fetch data from a specified MongoDB collection.

    SAP RFC Input

    Allows you to call functions that have been developed in the SAP system through RFC APIs to fetch data.

    Elasticsearch Input

    Allows you to fetch data from a specified Elasticsearch database.

    Dataset Input

    Allows you to fetch data from file datasets (Excel, TXT, XML, and CSV), tree datasets, stored procedures, programs, built-in datasets, and associated datasets. Among them, stored procedures, programs, built-in datasets, and associated datasets can only be defined in FineReport Designer.

    iconNote:
    The Dataset Input operator cannot fetch column values containing negative infinity (-∞) or positive infinity (+∞) from FineBI Public Data.

    Data Output

    DB Table Output

    Allows you to output data into tables in relational databases.

    Parameter Output

    Allows you to output obtained data as parameters for downstream nodes to use.

    API Output

    Allows you to output data into APIs.

    Jodoo Output

    Allows you to output data into Jodoo forms.

    File Output

    Allows you to output data as files.

    MongoDB Output

    Allows you to output data into the MongoDB database.

    Elasticsearch Output

    Allows you to output data into Elasticsearch.

    Connection

    Data Association

    Allows you to associate two tables from different source databases to generate a new table. The join methods are as follows:

    • Left Join: Returns all records from the left table and the records where the join condition is met in the right table.

    • Right Join: Returns all records from the right table and the records where the join condition is met in the left table.

    • Inner Join: Returns only the records where the join condition is met in the two tables.

    • Full Outer Join: Returns all records from both tables based on the join condition.

    Data Comparison

    Allows you to compare two inputs and get the new, deleted, same, or updated data.

    Union All

    Allows you to concatenate multiple tables row-wise and output a combined table.

    Transformation

    Column to Row

    Allows you to convert the columns in a data table to rows.

    Row to Column

    Allows you to convert the rows in a data table to columns.

    JSON Parsing

    Allows you to parse JSON data and output data in a row-column format.

    XML Parsing

    Allows you to parse XML data and output data in a row-column format.

    Field Setting

    Allows you to select and rename fields, and change data types.

    New Calculation Column

    Allows you to obtain a new field by referencing or calculating the original fields without affecting them.

    Data Filtering

    Allows you to filter required data records.

    JSON Generation

    Allows you to select fields and convert table data into multiple JSON objects (which can be nested).

    Field-to-Column Splitting

    Allows you to split field values according to specific rules (delimiters or the number of characters), where the split values form multiple new columns.

    Field-to-Row Splitting

    Allows you to split field values according to specific rules (delimiters), where the split values form a new column.

    Group Summary

    Allows you to group data based on certain criteria and perform summary calculations on the grouped data.

    Laboratory

    Spark SQL

    Allows you to query and process data using the built-in Spark calculation engine where parameters and functions are supported.

    Python

    Allows you to call Python scripts to perform complex data processing.

    Others

    Remark

    Allows you to add remarks to tasks and nodes.

     

    附件列表


    主题: Data Development - Scheduled Task
    Previous
    Next
    • Helpful
    • Not helpful
    • Only read

    滑鼠選中內容,快速回饋問題

    滑鼠選中存在疑惑的內容,即可快速回饋問題,我們將會跟進處理。

    不再提示

    10s後關閉

    Get
    Help
    Online Support
    Professional technical support is provided to quickly help you solve problems.
    Online support is available from 9:00-12:00 and 13:30-17:30 on weekdays.
    Page Feedback
    You can provide suggestions and feedback for the current web page.
    Pre-Sales Consultation
    Business Consultation
    Business: international@fanruan.com
    Support: support@fanruan.com
    Page Feedback
    *Problem Type
    Cannot be empty
    Problem Description
    0/1000
    Cannot be empty

    Submitted successfully

    Network busy