Example of Scheduled Task Invocation

Application Scenario

As mentioned in the Fetching Data Using Loops with a Spark SQL Data Sequence, you can use the constructed date sequence to fetch data via API.

Using Loop Container to fetch data from an API can be considered the smallest scheduling unit, which applies to tasks with multiple preceding Parameter Output nodes and different scheduling cycles. Creating this unit in every task is overly complex.

Implementation Method

You can set the scheduling unit (fetching data from an API using Loop Container) as a subtask, and invoke this subtask using the Scheduled Task Invocation node in other tasks.

For example, in the "Creating a Date Sequence" section of the document "Fetching Data Using Loops with a Spark SQL Data Sequence", a date sequence has been created, with which you can create a scheduled task as the subtask that fetches data with Loop Container.

Since the invoked task needs to be executed cyclically, and Loop Container requires a traversal parameter as input, the already created date sequence parameters cannot be used. You need to reconfigure the date sequence parameters in the new scheduled task to facilitate traversal by Loop Container.

Obtaining Parameters Passed by the Parent Task

First, create a task and build the date sequence referring to the "Setting the Start and End Time of the Date Sequence" and "Creating a Date Sequence" sections of the document "Fetching Data Using Loops with a Spark SQL Data Sequence."

Then, create a scheduled task as a subtask and add a Data Transformation node. Use the Spark SQL operator to obtain parameters passed by the parent task and process them into a multi-row array using functions, as shown in the following figure.

Note:

Since there is no direct upstream and downstream connection between the invoked task and the Scheduled Task Invocation node, the data preview will fail due to parameter loading failure. However, you don't need to worry, as this does not affect the actual execution of the task.

The statement in Spark SQL is as follows:

select explode(split('${loop_container}', ',')) as loop_container

Output the value as a parameter, and name the parameter the same as the date sequence parameter in the section "Creating a Date Sequence" of the document "Fetching Data Using Loops with a Spark SQL Data Sequence", as shown in the following figure.

Configuring a Loop Container Node

Refer to section "Using a Loop Container Node to Loop Through Parameter Values" of the document "Fetching Data Using Loops with a Spark SQL Data Sequence" to confugure nodes within the Loop Container node, as shown in the following figure.

Invoking the Task

Save the subtask, enter the parent task, add a Scheduled Task Invocation node after the nodes creating the date sequence, and select the created subtask as the task to be invoked. Since the subtask needs the date sequence parameter, you should tick the option Pass Parameter in Current Task to Subtask and select the date sequence parameter, as shown in the following figure.

Execute the task after completing the configuration.

Effect Display

Click Run to run the task. The data fetched from the API daily according to the date sequence is displayed. Meanwhile, the specific user will be notified of the result of each loop, as shown in the following figure.