Shell Script

Version Description

Application Scenario

You want to invoke Kettle tasks, Python-based computing tasks, and Java/Spark programs, and back up and restore databases in FineDataLink.

Function Description

You can run shell scripts on a specified server connected using the Shell Script node to execute external independent data processing tasks.

For example, you can execute SVN updates, manage and clean files, invoke Kettle tasks, Python-based computing tasks, and Spark-based computing tasks, and back up and restore databases. Also, you can use parameters set in FineDataLink in the shell script.

Restriction on Use

Only scripts with the .sh extension on Linux systems can be executed.

Configuring the SSH Protocol Data Connection

To execute a specified Shell script on the target server, you must first establish an SSH connection to the server. For details about configuring the SSH data connection, see SSH Data Connection.

Preparing the Shell Script

Take the following shell script as an example.

$0 refers to the shell script name. $1, $2, and $3 refer to the parameters (with values assigned in FineDataLink) to be written into the script.

After executing the script, you get three output parameter values, where $1 returns the data (which is Ketchup in this example) stored at the specified location in the FDL_PRODUCTS table, $2 returns the task execution time, and $3 returns the manually assigned value b3, as shown in the following figure.

Setting Parameters

Use the Parameter Assignment node to fetch the $1 parameter from the data table.

Drag in a Parameter Assignment node and fetch the data to be used as the parameter (which is Ketchup in this example) from the data table, as shown in the following figure.

Data in the FDL_PRODUCTS table is shown in the following figure.

Add $2 and $3 parameters in Parameter List, as shown in the following figure.

Setting the Shell Script Node

Drag in a Shell Script node, connect it to the Parameter Assignment node, select the SSH connection with the target remote server where the task is executed, enter the script path, click Add Parameter, and add b1, b2, and b3 parameters set in the section "Setting Parameters," as shown in the following figure.

Note:

When a Shell script in a loop container loops through a parameter, the value fetched corresponds to that in the ongoing loop.

Executing the Task

Click Run to execute the task, as shown in the following figure.

You can set Log Level Setting to INFO to get the detailed log, which is shown in the following figure.

After successful execution, you can see the output parameter values in the output file set in the script, as shown in the following figure.

Judging the Shell Script Execution Status

For details, see Description of Shell Script Execution Results.

Subsequent Operations

You can set a scheduling plan for the task to execute the shell script periodically. For details, see Scheduling Plan.

You can use the following three methods in shell scripts to output information. 

• Use the echo command: Output information to standard output.

• Use the printf command: Output formatted text. It is similar to the print function in other programming languages.

• Use the echo command with redirection: Redirect the output result to a file or another program.

Here are example codes of each command. 

Use the echo command to output text:

echo "Hello, World!"

Use the printf command to output formatted text:

printf "Name: %s\nAge: %d\n" "Alice" 30

Redirect the output to save it to a file:

echo "This will be saved to a file" > output.txt

The echo and printf commands output information in the log, as shown in the following figure.

The information is written to a file without being shown in the log when the echo command with redirection is used.