A company needs to fetch all ticket data from a business system for analysis.
Due to the high data volume, it is unfeasible to fetch all data in a single request. Parameters must be used.
In the following API document, pageIndex represents the page number of data, and pageSize represents the number of data records per page.
Obtain the total number of pages from the API, and output a parameter total_pages with the total page count as its value, which determines the number of iterations required.
Enable paginated data fetch with the page number mode. Use the parameter ${pageNum} with an initial value of 1 and an interval of 1. This ensures that the first execution fetches data from the first page, and each subsequent execution increments the page number by 1 to fetch data from the next page. Set the pagination end condition to ${pageNum} > ${total_pages}.
Set the pageSize value to 100, meaning each page is limited to 100 records. Check the total page count calculated based on this page size in the return value.
Add a Parameter Assignment node, configure the API, and add the parameter pageSize with a value 100, as shown in the following figure.
Outputting the Total Page Count as a Parameter
Output a parameter with the total page count as its value, which will be used to form the pagination end condition, enabling the page-by-page data fetching to stop after the final page is reached.
The total page count is returned in the API response, as shown in the following figure.
Click Output Parameter, set the parameter name to total_pages and the value source to the TotalPages column, as shown in the following figure.
Click Parameter Preview. The total number of pages is 101 in this example, meaning there are 101 pages of data, 100 records per page.
Configuring Paginated Data Fetching
Add a Data Synchronization node, set Data Connection to API Input, enable Paginated Data Fetch, set Pagination Method to Page Number, and set Update Strategy to Initial Value 1, Interval 1. This means that, according to the set update strategy, the process will start from Page 1 and increment the ${pageNum} value to fetch data from the API page by page, as shown in the following figure.
Pagination ends when the ${pageNum} value increments to a value greater than the ${total_pages} value.
Select the request method and input the API URL. Add the parameter pageSize with the value 100, and the parameter pageIndex with the value ${pageNum}. This means that with 100 records per page, the process starts from Page 1 and increments the page number to fetch data from the API, as shown in the following figure.
Since the response is in JSON format, you can extract specific data using a JSON path, as shown in the following figure.
Set the target database and table on the Data Destination and Mapping tab page, as shown in the following figure.
Effect Display
Save and run the task. You can view all ticket data fetched from the API in the database table, as shown in the following figure.
Use the Loop Container node and the built-in parameter loopTimes. On the first iteration, the pageIndex value is 1, and then the value increments by 1 for the loop to fetch data from all pages. The loop continues while the loopTimes value is less than or equal to the total_pages value and stops when the loopTimes value is greater than the total_pages value.
Obtaining the Total Page Count of Ticket Data
Fetching All Ticket Data by Incrementing the Page Number
loopTimes is a built-in parameter of Loop Container. It represents the current loop count within the container, starting from 1 and incrementing by 1 with each subsequent iteration. This parameter can also be used to fetch all data from the API page by page.
Add a Loop Container node, as shown in the following figure.
Drag a Data Synchronization node into Loop Container, as shown in the following figure.
Select the request method and input the API URL. Add the parameter pageSize with the value 100, and the parameter pageIndex with the value ${loopTimes}. This means that with 100 records per page, the process starts from Page 1 and increments the page number to fetch data from the API, as shown in the following figure.
If the response is complex, you are advised to use the JSON Parsing operator in a Data Transformation node.
Click Loop Container, set the loop limit, and set Loop Type to Do-While Loop. Click Configure Condition and set the condition to ${loopTimes}<=${total_pages}, as shown in the following figure.
This ensures that the loop continues until the page number exceeds the total page count, thereby fetching all ticket data.
滑鼠選中內容，快速回饋問題
滑鼠選中存在疑惑的內容，即可快速回饋問題，我們將會跟進處理。
不再提示
10s後關閉
Submitted successfully
Network busy