Overview
Version
FineDataLink Version | Functional Change |
4.0.19 | Added a Data Filtering operator, allowing you to filter data. |
4.0.28 | Allowed you to get data whose value consisted entirely of spaces through filtering by selecting the Null operator. |
Application Scenario
Data contains null values, and you want to filter out the null data.
The data table contains product contract records from multiple years and you want contract data from 2017 only.
You can use the Data Filtering operator to achieve that.
Function Overview
You can use the Data Filtering operator to obtain the required data through filtering, as shown in the following figure.
Notes
The date to be filtered should be in the yyyy-mm-dd HH:mm:ss format. If the data to be filtered is in the yyyy-mm-dd format, transform it to data in the yyyy-mm-dd HH:mm:ss format or use parameters.
Function Introduction
Setting Item Description
Setting Item | Description |
Field |
|
Operator | The supported operator varies with field types. |
Fixed value input box | The value input box such as the date input box and the string input box varies with operators, where you can enter ${Parameter name} to reference a parameter. You do not need to enclose the value in the fixed value input box in single or double quotes. If you select In or Not In as the operator, separate multiple values by comma. The Contain operator corresponds to the SQL LIKE clause and supports single-value filtering only. For multi-value filtering, use the In and Not In operators. For example
|
Remark:
| |
Operator Description
Note:
1. In cases where the field type is either text or value, Field Setting offers an In operator, which encompasses both the Equal to operator (=) and the In operator (∈).
2. When you select the Null operator, you can get data with empty values, null values (case-insensitive), and empty strings ("") through filtering. Starting from FineDataLink 4.0.28, you can additionally get data with space values.
3. Starting from FineDataLink 4.1.1, the conditional judgment logic has been standardized. For details, see Judgment Logic Description
Field Type | Selectable Operator |
Text | In and Not in Contain, Not Contain, Begin with, End with, Not Begin with, Not End with, Null, and Not Null |
Others | Equal to and Not Equal to |
Value | In and Not in Between, Not Between, Greater Than, Less Than, Greater Than or Equal to, Less Than or Equal to, Null, and Not Null |
Date | Between and Not Between Before, After, Equal to, Not Equal to, Null, and Not Null |
Example
You can download the example data:SOrder2.xls
This document aims to filter data where the OrderDate is 2011 and the Freight is greater than 100.
Data Input
Create a scheduled task.
Add a Data Transformation node.
Click the Data Transformation node, add a DB Table Input operator, and get data in the SOrder2 table.
Field Setting
Since the fields in this table are all of the text type, you need to change date-related fields and value-related fields to date-type and value-type fields, respectively.
Note: Skip this step if the field types are correct.Add a Field Setting operator, connect it to the DB Table Input operator, and modify the field type.
Data Filtering
Set filter conditions to fetch data where the OrderDate is 2011 and the Freight is greater than 100.
Click Data Preview to view the fetched data, as shown in the following figure.
Data Output
Add a DB Table Output operator to output the data after filtering.
Running the Task
Click Run in the upper right corner. After successful execution, a message indicating successful execution will be displayed in Log.
You can view the data obtained through filtering in the OrderAfter20110101 table.
Best Practice
Data Filtering enables fuzzy query and filtering of fields without complex SQL statements, improving data processing efficiency. For details, see.Fuzzy Field Querying/Filtering
Previous：Spark SQL
