API Input - Paginated Data Fetching

4.1.3

Data Fetch Limit

It restricts the number of paginated data fetches.

• The value is 128 by default.

• It cannot be empty. The maximum value is 10000.

• If the limit is reached before the pagination end condition is met, the paginated data fetch will be terminated and an error will be reported.

Pagination Method - Page Number

Parameter:

• As an operator-level parameter, pageNum is only valid for a single API call and has the highest priority among parameters with the same name.

• The value of pageNum is incremented after the completion of a single API call.

• You can use the pageNum parameter in various formats in Query Parameter and Body. Its usage is the same as that of regular parameters.

Update Strategy:

• Initial value: You can configure the initial value, which is 1 by default and cannot be empty.

• Interval: You cannot modify the interval, which is 1 by default and cannot be empty.

If the initial value and the interval are configured, the parameter value will be incremented after each loop, increasing by the value of Interval.

Pagination End Condition:

• httpStatus: Set the condition based on the HTTP status code returned by the API during the current loop.

• JSONPath: Fill in the JSONPath expression based on the body in the response of this loop. (Only a single value can be returned. If an array or a JSON object is returned, it will become a string.) If the JSONPath of the response body does not exist, the value will be empty.

• requestTimes: Set the condition based on the number of paginated requests. If the API Input operator is in a loop container, the number of paginated requests will be counted separately for each loop of that container.

• pageNum: Set the condition based on the page number parameter $pageNum.

Note:

Referencing parameters is supported.

Pagination Method - Offset

Parameter:

• As an operator-level parameter, Offset is only valid for a single API call and has the highest priority among parameters with the same name.

• The value of Offset is incremented after the completion of a single API call.

• You can use the Offset parameter in various formats in Query Parameter and Body. Its usage is the same as that of regular parameters.

Update Strategy:

• Initial value: You can configure the initial value, which is 1 by default and cannot be empty.

• Interval: You can modify the interval, which is 20 by default and cannot be empty.

If the initial value and the interval are configured, the parameter value will be incremented after each loop, increasing by the value of Interval.

Pagination End Condition:

• httpStatus: Set the condition based on the HTTP status code returned by the API during the current loop.

• JSONPath: Fill in the JSONPath expression based on the body in the response of this loop. (Only a single value can be returned. If an array or a JSON object is returned, it will become a string). If the JSONPath does not exist in the response body, the value will be empty.

• requestTimes: Set the condition based on the number of paginated requests. If the API Input operator is in a loop container, the number of paginated requests will be counted separately for each loop of that container.

• offset: Set the condition based on the offset parameter $offset.

Note:

Referencing parameters is supported.

Pagination Method - Cursor

Cursor:

• As an operator-level parameter, cursor is only valid for a single API call and has the highest priority among parameters with the same name.

• It is used in scenarios where the results of the current loop are passed to the next one.

• You can use the cursor parameter in various formats in Query Parameter and Body. Its usage is the same as that of regular parameters.

Update Strategy:

• Initial value: It is empty by default. You can enter a value or leave it empty. Referencing parameters is supported.

• Update value:

• If the JSONPath does not exist in the response body, the value will be empty.

• Fill in the JSONPath expression based on the body in the response of this loop. (Only a single value can be returned. If an array or a JSON object is returned, it will become a string).

Note:

The cursor parameter is updated only when the pagination end condition is not met.

Pagination End Condition:

• httpStatus: Set the condition based on the HTTP status code returned by the API during the current loop.

• JSONPath: Fill in the JSONPath expression based on the body in the response of this loop. (Only a single value can be returned. If an array or a JSON object is returned, it will become a string). If the JSONPath does not exist in the response body, the value will be empty.

• requestTimes: Set the condition based on the number of paginated requests. If the API Input operator is in a loop container, the number of paginated requests will be counted separately for each loop of that container.

• cursor: Set the condition based on the cursor parameter $cursor.

Note:

Referencing parameters is supported.

Retry

If a single API call during the paginated data fetch fails, the entire Data Transformation node will be retried.

Call Frequency Control

The frequency of API calls during the paginated data fetch in one execution instance is controlled by Call Frequency Control.

• Two Configuration Items of Response Body Processing: JSON Path 

• Expand Parsed JSON Data into a Two-Dimensional Table

• If you have only enabled Paginated Data Fetch, the JSON data obtained from each fetch will be assembled into a two-dimensional table with one responseBody column and multiple rows.

• If you have set the JSONPath expression in Response Body Processing and enabled Paginated Data Fetch, the JSON data of the corresponding path obtained each time will be assembled into a two-dimensional table with one responseBody column and multiple rows.

The JSON data obtained each time may be different. If the corresponding JSON path cannot be found, the value will be empty.

• If you have set the JSONPath expression to $.data in Response Body Processing and enabled Expand Parsed JSON Data into a Two-Dimensional Table and Paginated Data Fetch, the JSON data under the corresponding path obtained each time will be split into columns and assembled into a two-dimensional table with multiple columns and multiple rows.

The JSON data obtained each time may be different. Each time the JSON data is expanded into a two-dimensional table. A column will be created for each new attribute.

The relevant column value of data that does not have the attribute will be empty.

Return Value Format

If the response body is in XML format, paginated data fetch is not supported.

Information Type (Parameter Assignment)

• Response Body

• Response Header

• Status Code

If the response body, response header, and status code are parsed by the Parameter Assignment node at the same time, the response obtained each time will be assembled into a two-dimensional table with multiple columns and multiple rows.