Overview
Version
| FineDataLink Version | Functional Change |
|---|---|
| 1.5 | Supported the API data source. |
| 4.1.3 | Supported paginated data fetching. For details, see API Input - Paginated Data Fetching. |
| 4.1.7.3 | Allowed specifying Response Encoding in Advanced Configuration. For details, see Data Synchronization - API. |
Application Scenario
You want to further process data fetched from an API, which cannot be achieved using the API Input function in Data Synchronization.
Function Description
The API Input operator is added to the Data Transformation node, allowing you to fetch data from an API and process it using other operators, as shown in the following figure.
Note:1. If the API return value is simple JSON data that requires no complex parsing, you can use the API Input function in Data Synchronization.
2. You can use parameters when fetching data from an API using the API Input operator. Dynamic parameter values are supported. For details about where to use parameters, see the "Function Description" section of this document. For details about specific examples, see the "Practical Application of Fetching Data from APIs" section.
Prerequisite
1. For details about APIs, see Getting Started with APIs.
2. Before fetching data from an API, test the API using Postman or other debugging tools and ensure availability, as shown in the following figure.
3. The API Protocol (Timed) function point has been registered.
Use Restriction
1. The default request timeout is 10 seconds in FineDataLink. Any API query that exceeds this duration will fail.
In case of large data volumes that cause long request time, you can use the FDLTimeout field in Headers when fetching data from an API to prolong the timeout.
Note that the parameter name is FDLTimeout (case-sensitive). Set the parameter value to a positive integer, as shown in the following figure. The unit is second (s).
Note:
2. You can parse the JSON data fetched from the API in Return Value Processing, which provides limited parsing ability compared with the JSON Parsing operator. The difference is shown in the following table.
| Comparison Item | Difference |
|---|---|
| JSON Parsing operator | 1. The JSON Parsing operator allows parsing multiple fields simultaneously, while Return Value Processing only supports single-field parsing. 2. You can generate only one parsing path with Expand Parsed JSON Data into a Two-Dimensional Table in Return Value Processing ticked, while the JSON Parsing operator allows generating multiple paths. |
| Return Value Processing setting item |
Therefore, the API return value in simple JSON format that requires no complex parsing can be parsed in Return Value Processing. Otherwise, you are advised to use the JSON Parsing operator.
Function Description
The setting page of API-based data fetching is shown in the following figure.
Note:1. The data fetched from an API can be output as parameters (using the Parameter Output operator, the Parameter Assignment node, and the Assignment Parameter function in Global Parameter) or to databases, APIs, file systems, and other locations.
2. When fetching data from HTTPS-based APIs, you can cancel certificate validity verification by adding an SSL-certificate-verification parameter in Headers and setting its value to false. For details, see Logic Description of API Output.
Setting Item Description
| Letter | Item | Description | ||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| A | Request Method | FineDataLink supports both GET and POST requests. For details about API instructions, see Getting Started with APIs. Note: | ||||||||||||||||||||
| B | API URL | Enter the URL of the API to be called. | ||||||||||||||||||||
| C | Request Parameter (Query Parameter) |
| ||||||||||||||||||||
| D | Authorization (Authentication Method) |
| ||||||||||||||||||||
| E | Headers (Header Parameter) |
| ||||||||||||||||||||
| F | Body (Request Content) | Note: The supported types of the request body in the raw format include TEXT, JSON, and XML. The body syntax is as follows.
The formats are GET:"Parameter name 1=Parameter Value&Parameter name 2=Parameter Value" and POST:{"Parameter name 1":Parameter Value,"Parameter name 2"=Parameter Value}. | ||||||||||||||||||||
| G | TLS/SSL (Self-Signed Certificate) | APIs for which HTTPS is enabled support self-signed certificates. After a self-signed certificate is configured, HTTPS requests are handled according to the SSL/TLS protocol. Note:
| ||||||||||||||||||||
| H | Return Value Processing (The JSON format response content can be further processed.) | By default, the first-level node is parsed if you do not enter the JSONPath expression. You can customize the parsing hierarchy in the format of $.Parent node name.Child node name, such as $.store (one level) and $.store.book (two levels). For details, see the "Return Value Processing" section of this document. | ||||||||||||||||||||
| I | Call Frequency Control | You can limit the number of calls of the current API in a single execution instance by configuring Call Frequency Control. You can control the frequency at the second/minute/hour/day level.
For example, the API with the call frequency of Every Second Execute 1 Time(s) will be called once a minute after you run the task, which can be found in the log.
| ||||||||||||||||||||
| J | Paginated Data Fetch | For details, see API Input - Paginated Data Fetching. | ||||||||||||||||||||
| K | Response Encoding | You can specify the response encoding. Supported character encoding methods include UTF-8, UTF-16, and GBK. The API response is parsed according to response encoding. |
Return Value Processing
If the API return value is in the JSON format and you want to extract a specified JSON array, you can parse JSON data in Return Value Processing.
The following table lists the correspondence between data types.
| Type | Data Type |
|---|---|
| Integer | Long and int |
| String | String |
| Floating point | Double and float |
| Boolean | Boolean |
| Date and time | Date |
| Others | Empty |
Take the API https://demo.finedatalink.com/help/book.json as an example.
Default Return Value
By default, the result of API-based data fetching includes all objects, with the first-level node being parsed.
This means the store, store.book, store.bicycle, store.bicycle.color, and store.bicycle.price data is returned.
Specifying the Return Value
You can customize a JSON path to obtain a specific JSON object or array from the result.
Customize the parsing hierarchy using the Parent node name.Child node name format, such as store (one level), store.book (two levels), and store.book.author (three levels).
For instance, you can enter $.store.book in the path to fetch all data in the book array from the store object, as shown in the following figure.
By entering $.store in the path, you can fetch the first-level node data in the store object, including book, bicycle, bicycle.color, and bicycle.price data, as shown in the following figure.
For details about JSONPath expressions, see JSONPath expression.
Combined Use with Parameters
Description
When fetching data from an API, you can reference parameters in the following setting items.
| Setting Item | Description | Remark |
|---|---|---|
| Query Parameter (marked with Letter C in the figure in the "Function Description" section) | You can reference parameters in the input box of Parameter Value. | For details about parameter configuration and use in FineDataLink, see Parameter Configuration and Use. |
| Authentication Method (D) | Bearer Token: You can reference parameters in the input box of Token. Other: You can reference parameters in the input box of Authorization. | |
| Header Parameter (E) | You can reference parameters in the input box of Parameter Value. | |
| Body (F) | You can reference parameters in the body input box. |
The data fetched from APIs can be output as parameters, as described in the following table.
| Description |
|---|
You can output the data fetched from the API as parameters using the Parameter Assignment nodes, the Parameter Output operator, and the Assignment Parameter function in Global Parameter, where the API response headers, response body, and status code can be obtained. 1. Application scenarios of Assignment Parameter and Parameter Assignment: a. The API return value is in a simple JSON format and requires no complex parsing. b. The difference between the Assignment Parameter function in Global Parameter and the Parameter Assignment node is that the parameter output by Assignment Parameter can be used by multiple scheduled tasks, while the parameter output by the Parameter Assignment node can only be used by its downstream nodes. 2. Application scenarios of Parameter Output: a. The data fetched from the API needs complex processing before being output as parameters. b. The general procedure is as follows:
|
For details about referencing parameters when fetching data from an API, see Parameter Configuration and Use.
Notes
1. You are advised to set default values for parameters when referencing parameters during API-based data fetching and outputting API data as parameters, to facilitate effect viewing at nodes and operators where parameters are referenced. For dynamic parameter values (such as those only valid for two minutes), you are advised to update the default parameter value before viewing the effect at subsequent nodes.
Note:2. To use a parameter when calling an API:
Ensure that null values are filtered out before calling the API, as no data can be fetched from the API if a null parameter value exists.
You are advised to use the encoding, encryption, and decryption functions in Spark SQL if the required parameter needs to be encrypted.
You are advised to concatenate values using the Spark SQL operator or functions (in SQL Script, New Calculation Column, and operators) if signature values need to be generated dynamically.
Practical Application of API-Based Data Fetching
For details about API-based data fetching, see Instructions on API-Based Data Fetching.
For details about examples of API-based data fetching, see Overview of API-Based Data Fetching.
Previous：API Input - Paginated Data Fetching
