Instructions on API-Based Data Fetching

FineDataLink provides two functions to fetch data from APIs, with the following differences: 

This document describes how to use the two functions in practical scenarios.

Note:

The value of the parameter output by the Parameter Assignment node or the Parameter Output operator with enabled Closure Character is enclosed with single quotation marks by default. If you enclose the parameter with single or double quotation marks to reference it during API-based data fetching, data fetching may fail. Therefore, enable Closure Character according to actual conditions.

Fetching Data from the API and Saving It

Scenario Description

The data fetched from the API requires simple parsing.

Implementation Method

Method One:

Use the API Input function in Data Synchronization to get the required data, as shown in the following figure.

Method Two:

Use the API Input operator to read data from the API, then parse data using the JSON Parsing or the XML Parsing operator,  and finally, output the data. (You can use other data processing operators to further process the data before outputting it.)

Example

For details, see Data Synchronization - API, XML Parsing, Executing the Scheduled Task on Weekdays Only, and Calling Scheduled Task Related APIs.

Fetching Data from the API and Outputting it as a Parameter

Application Scenario

Read data from the API and output it as a parameter for it to be used by subsequent nodes/operators.

Implementation Method

You can output the data fetched from the API as a parameter using the Parameter Assignment nodes, the Parameter Output operator, and the Assignment Parameter function in Global Parameter, where the API response headers, response body, and status code can be obtained.

For details, see API Input.

Example

You want to call an API named Get_Department_List to obtain all department IDs. Calling Get_Department_List requires using the access_token parameter, which can be obtained by calling another API named Get_Access_Token.

Therefore, you need to call Get_Access_Token to obtain the access token and output it as a parameter, and use the parameter to call Get_Department_List.

Scenario Description

You can use the paginated data fetching scheme when the API contains pagination-related parameters (such as pageIndex, pageSize, and pagination parameters). 

Implementation Method

Note:

If a page contains no data, the data fetched from it is empty during paginated data fetching, which may cause a task execution error.

Method One:

FineDataLink provides a Paginated Data Fetch function. For details, see API Input - Paginated Data Fetching.

Method Two:

Using API Input in combination with the Loop Container node. You are advised to learn how to use Loop Container and then study documents introducing paginated data fetching.

Example

For details, see Overview of API-Based Data Fetching.

For details about fetching data from an API using both Loop Container and pagination methods, see Fetching Data Using a Nested Loop.

Application Scenario

You need to fetch data from an API that accepts only a single value for a given parameter at one time. To facilitate data fetching from the API, you can obtain the parameter values used to call the API, output them to a new parameter, and use the Loop Container node to traverse all parameter values.

Note:

Set Loop Type to For-Each Loop or Do-While Loop according to the actual scenario. For details, see Description of Loop Container Logic.

Implementation Method

Use the API input and the parameter functions in combination with the Loop Container node. The following is an example.

You want to call the Get_Department_Member API to obtain the user IDs, which requires using the parameters Access_Token and Department_ID. The Get_Department_Member API accepts only a single value for a parameter at a time. To facilitate data fetching, you can:

1. Call the Get_Access_Token API to obtain the access token and output it to the Access_Token parameter. (The parameter value is dynamic.)

2. Call the Get_Department_List API to obtain all department IDs and output them to the Department_ID parameter.

3. Use the Access_Token and Department_ID parameters in a Loop Container node for it to iterate through all access tokens and department IDs to fetch data.

Application Scenario

You want to encrypt the API calls with encrypted parameters. For example, calling the Get_Ticket_List API requires using the signature parameter sign, and the request is accepted only when the sign value is valid.

Implementation Method

Use the encryption function to encrypt the parameter in the Spark SQL operator. For details, see Encryption Function.