When using scheduled tasks for data development, you often need to preview the data to check if the calculation results of the tasks are correct.
FineDataLink provides the Sample Setting function in input operators added in the Data Transformation node, which allows setting the number of data calculation samples participating in a preview for result validation.
In FineDataLink, you can preview the first 1000 rows of data by default in the Data Synchronization, Parameter Assignment, and Data Transformation nodes. Field types are also displayed on the Data Preview page, facilitating you in data processing and calculation.
The Sample Setting and Data Preview functions improve your efficiency in data development and debugging, enabling you to identify data issues as early as possible during the task design stage.
If the data volume is too large (for example, 10 million rows), you are not advised to set Sample Volume to Full Volume in Sample Setting, as this may cause the frontend to become unresponsive.
You can download the example data: DEMO_Contract.xlsx and Customer.xls.
You need to associate the data of the two files using the Customer ID field to add the customer data to the Demo_Contract table.
Input data using the File Input operators, and perform data calculations using the Spark SQL operator, as shown in the following figure.
Since the Customer table contains a large volume of data, Number of Data Rows in Sample Setting is set to 5000 by default, meaning that 5000 rows of data are used for calculations on the Data Preview page.After data association, the data beyond the 5000th row in the Customer table does not participate in the calculations, making it difficult to properly verify the correctness of the calculation results displayed in Data Preview, as shown in the following figure.
To avoid this situation, select the File Input operator named Customer, click Sample Setting, and set Sample Volume to Full Volume, thus making all the data participate in the calculation during data preview, as shown in the following figure.
The Data Preview page displays the calculation result using full-volume data, as shown in the following figure.
The following table describes the field types parsed from the database to FineDataLink.
Text
varchar
Numeric
int, long, float, and double
Date
date and timestamp
Others
binary and boolean
During task development, data input from the data source is first parsed. The parsed data is calculated based on the data volume set in Sample Setting and the calculation result is displayed in Data Preview.
The following table describes the logic for setting the parsing volume of data from different input sources.
Data Synchronization\Parameter Assignment
Dataset Input (parse the first row), API Input (parse the entire segment), and MongoDB Input (parse the first 100 rows)
Data Transformation
Dataset Input (parse the first row), API Input (parse the entire segment), MongoDB Input (parse the first 100 rows), and File Input (parse the first 5000 rows)
JSON Parsing (parse the first row of the selected fields) and XML Parsing (parse the first row of the selected fields)
滑鼠選中內容，快速回饋問題
滑鼠選中存在疑惑的內容，即可快速回饋問題，我們將會跟進處理。
不再提示
10s後關閉
Submitted successfully
Network busy