You want to further process data fetched from an API, which cannot be achieved using the API Input function in Data Synchronization.
The API Input operator is added to the Data Transformation node, allowing you to fetch data from an API and process it using other operators, as shown in the following figure.
1. If the API return value is simple JSON data that requires no complex parsing, you can use the API Input function in Data Synchronization.
2. You can use parameters when fetching data from an API using the API Input operator. Dynamic parameter values are supported. For details about where to use parameters, see the "Function Description" section of this document. For details about specific examples, see the "Practical Application of Fetching Data from APIs" section.
1. For details about APIs, see Getting Started with APIs.
2. Before fetching data from an API, test the API using Postman or other debugging tools and ensure availability, as shown in the following figure.
3. The API Protocol (Timed) function point has been registered.
1. The default request timeout is 10 seconds in FineDataLink. Any API query that exceeds this duration will fail.
In case of large data volumes that cause long request time, you can use the FDLTimeout field in Headers when fetching data from an API to prolong the timeout.
Note that the parameter name is FDLTimeout (case-sensitive). Set the parameter value to a positive integer, as shown in the following figure. The unit is second (s).
2. You can parse the JSON data fetched from the API in Return Value Processing, which provides limited parsing ability compared with the JSON Parsing operator. The difference is shown in the following table.
2. You can generate only one parsing path with Expand Parsed JSON Data into a Two-Dimensional Table in Return Value Processing ticked, while the JSON Parsing operator allows generating multiple paths.
Therefore, the API return value in simple JSON format that requires no complex parsing can be parsed in Return Value Processing. Otherwise, you are advised to use the JSON Parsing operator.
The setting page of API-based data fetching is shown in the following figure.
1. The data fetched from an API can be output as parameters (using the Parameter Output operator, the Parameter Assignment node, and the Assignment Parameter function in Global Parameter) or to databases, APIs, file systems, and other locations.
2. When fetching data from HTTPS-based APIs, you can cancel certificate validity verification by adding an SSL-certificate-verification parameter in Headers and setting its value to false. For details, see Logic Description of API Output.
FineDataLink supports both GET and POST requests. For details about API instructions, see Getting Started with APIs.
Enter the URL of the API to be called.
You can add a parameter and enter the parameter name and value.
You can also suffix the URL with the parameter, as shown in the following figure.
No Authentication: It is the default value.
Bearer Token: Enter the token for authentication.
Basic Auth: Enter the username and password for authentication.
Other: Enter the Authorization value. Referencing parameters is supported.
The system supports various media types such as application/x-www-form-urlencoded, application/json, application/xml, multipart/form-data, text/xml, and text/plain.
You can enter a content type not in the drop-down list of Parameter Value in Headers.
The linkage between settings in Body and the Content-Type value is described in the following table.
application/xml
If the entered Content-Type value is beyond the linkage range, the settings in Body remain unchanged until the Content-Type value is changed.
If you enter a Content-Type value with no corresponding Body format/type and then modify the settings in Body, the Content-Type value in Headers will be overwritten with the type in Body.
The supported types of the request body in the raw format include TEXT, JSON, and XML.
The body syntax is as follows.
GET: Parameter name 1=Parameter Value&Parameter name 2=Parameter Value
POST: JSON parameter
The formats are GET:"Parameter name 1=Parameter Value&Parameter name 2=Parameter Value" and POST:{"Parameter name 1":Parameter Value,"Parameter name 2"=Parameter Value}.
APIs for which HTTPS is enabled support self-signed certificates.
After a self-signed certificate is configured, HTTPS requests are handled according to the SSL/TLS protocol.
By default, the first-level node is parsed if you do not enter the JSONPath expression. You can customize the parsing hierarchy in the format of $.Parent node name.Child node name, such as $.store (one level) and $.store.book (two levels).
For details, see the "Return Value Processing" section of this document.
You can limit the number of calls of the current API in a single execution instance by configuring Call Frequency Control.
You can control the frequency at the second/minute/hour/day level.
For example, the API with the call frequency of Every Second Execute 1 Time(s) will be called once a minute after you run the task, which can be found in the log.
For details, see API Input - Paginated Data Fetching.
You can specify the response encoding.
Supported character encoding methods include UTF-8, UTF-16, and GBK.
The API response is parsed according to response encoding.
If the API return value is in the JSON format and you want to extract a specified JSON array, you can parse JSON data in Return Value Processing.
The following table lists the correspondence between data types.
Take the API https://demo.finedatalink.com/help/book.json as an example.
Default Return Value
By default, the result of API-based data fetching includes all objects, with the first-level node being parsed.
This means the store, store.book, store.bicycle, store.bicycle.color, and store.bicycle.price data is returned.
Specifying the Return Value
You can customize a JSON path to obtain a specific JSON object or array from the result.
Customize the parsing hierarchy using the Parent node name.Child node name format, such as store (one level), store.book (two levels), and store.book.author (three levels).
For instance, you can enter $.store.book in the path to fetch all data in the book array from the store object, as shown in the following figure.
By entering $.store in the path, you can fetch the first-level node data in the store object, including book, bicycle, bicycle.color, and bicycle.price data, as shown in the following figure.
For details about JSONPath expressions, see JSONPath expression.
Description
When fetching data from an API, you can reference parameters in the following setting items.
Bearer Token: You can reference parameters in the input box of Token.
Other: You can reference parameters in the input box of Authorization.
The data fetched from APIs can be output as parameters, as described in the following table.
You can output the data fetched from the API as parameters using the Parameter Assignment nodes, the Parameter Output operator, and the Assignment Parameter function in Global Parameter, where the API response headers, response body, and status code can be obtained.
1. Application scenarios of Assignment Parameter and Parameter Assignment:
a. The API return value is in a simple JSON format and requires no complex parsing.
b. The difference between the Assignment Parameter function in Global Parameter and the Parameter Assignment node is that the parameter output by Assignment Parameter can be used by multiple scheduled tasks, while the parameter output by the Parameter Assignment node can only be used by its downstream nodes.
2. Application scenarios of Parameter Output:
a. The data fetched from the API needs complex processing before being output as parameters.
b. The general procedure is as follows:
Use the API Input operator to fetch data from the API.
Use the JSON Parsing or the XML Parsing operator to parse the fetched data.
Use other operators to process data after parsing.
Use the Parameter Output operator to output data as the parameter.
For details about referencing parameters when fetching data from an API, see Parameter Configuration and Use.
Notes
1. You are advised to set default values for parameters when referencing parameters during API-based data fetching and outputting API data as parameters, to facilitate effect viewing at nodes and operators where parameters are referenced. For dynamic parameter values (such as those only valid for two minutes), you are advised to update the default parameter value before viewing the effect at subsequent nodes.
2. To use a parameter when calling an API:
Ensure that null values are filtered out before calling the API, as no data can be fetched from the API if a null parameter value exists.
You are advised to use the encoding, encryption, and decryption functions in Spark SQL if the required parameter needs to be encrypted.
You are advised to concatenate values using the Spark SQL operator or functions (in SQL Script, New Calculation Column, and operators) if signature values need to be generated dynamically.
For details about API-based data fetching, see Instructions on API-Based Data Fetching.
For details about examples of API-based data fetching, see Overview of API-Based Data Fetching.
滑鼠選中內容，快速回饋問題
滑鼠選中存在疑惑的內容，即可快速回饋問題，我們將會跟進處理。
不再提示
10s後關閉
Submitted successfully
Network busy