Overview of Scheduled Task

  • Last update: August 20, 2024
  • Overview

    Version

    FineDataLink VersionFunctional Change

    1.9

    /

    4.0.14

    Added the node usage explanation that would be displayed when you hover the cursor over a node in the node list, allowing you to know how to use it quickly.

    4.0.28

    Added the Python Script node, allowing you to directly call Python scripts.

    4.0.29

    Added the Bat Script node, allowing you to directly call Bat script files in remote Windows environments.

    4.1.5.5

    Added the File Transfer node, allowing you to download/upload files between APIs, local storage, and SFTP/FTP servers, thus realizing file movement.

    4.1.6.3

    A scheduled task cannot be edited by multiple users simultaneously.

    4.1.9.1

    Supported parameter autocompletion in the SQL statement input box under Data Synchronization > DB Table Input, Body under Data Synchronization > API, SQL Script, the SQL statement input box under Parameter Assignment > DB Table Input, the SQL statement input box in DB Table Input, Body in API Input, Spark SQL, and Python.

    4.1.9.3

    You can use Recycle Bin to restore deleted tasks.

    You can use the Kettle Call node.

    Function Description

    The Data Development module supports data extraction, transformation, and loading through the use of various nodes and operators on a visual page. It enables the automatic running of scheduled tasks through the Timed Scheduling function, helping you effortlessly build offline data warehouses and ensuring efficient and stable data production.

    iconNote: 

    For details about some concepts of Data Development, see the section "Data Development" of FineDataLink Glossary.

    Function Overview

    0.jpeg

    Type
    Description

    Data Source

    FineDataLink's Data Development module supports the processing and integration of data from various data sources.

    For details, see the section "Types of Data Sources Supported by Data Development" of Data Sources Supported by FineDataLink.

    Task Management

    You can store the designed business processes under Data Development > Scheduled Task.

    You can manage scheduled tasks using Folder.

    Task Development

    · Rich in capabilities

    You can use general synchronization nodes for cross-database data synchronization and data cleansing calculations.

    You can use various script nodes to call Python scripts, Shell scripts, and so on.

    You can use process nodes with general nodes for complex logic processing, such as conditional branch, loop execution, output result transfer, and other complex processes.

    · Easy to use

    You can quickly arrange a task by dragging and dropping nodes.

    An intelligent SQL statement editor is provided.

    For details about the node types supported by FineDataLink, see the section "Node Introduction."

    Task Schedule

    · Task Control

    You can enable Timeout Limit and Retry After Failure for tasks.

    You can configure Task Priority, Dirty Data Tolerance, Result Notification, and Log Level Setting.

    · Scheduling Plan

    You can set Scheduling Start Time and Execution Frequency.

    You can set scheduling dependencies for tasks, for example, executing Task C only after tasks A and B have been executed successfully.

    For details about task scheduling, see Overview of Task Control.

    Task O&M

    O&M personnel can flexibly schedule tasks and monitor the real-time running status, reducing the heavy workload.

    For details, see Scheduled Task O&M.

    Other

    · Open capability: Rich open APIs are provided for you to implement functions such as triggering scheduled tasks by external systems. For details, see Description of Scheduled Task-Related API Calling.

    · Permission control: You can control permissions on page-based functions and data access. For details, see Scheduled Task Management Permission.

    · A scheduled task cannot be edited by multiple users simultaneously in FineDataLink 4.1.6.3 and later versions.

    · You can restore and manage deleted tasks in Recycle Bin in FineDataLink 4.1.9.3 and later versions.


    Node Introduction

    FineDataLink's Data Development module provides various types of nodes that can be used in combination to meet your diverse data processing needs.

    A node is a basic unit of a scheduled task. Multiple nodes form an execution process after being connected by lines and further form a complete scheduled task. FineDataLink provides the following nodes:

     1.png

    Type
    DescriptionNode NameApplication Scenario
    General

    It is used for data synchronization.

    Data Synchronization

    Used to extract data from one database to another.

    It is used for complex data cleansing calculations.

    Data Transformation

    Used to perform operations such as data association and synchronous data deletion.

    It provides multiple ways to download/upload files.

    File Transfer

    Used to download/upload files from/to APIs, local servers, and SFTP/FTP servers, thus realizing file movement.

    Note: This node is used to synchronize files, not extracting the data in the files.


    For details about differences between Data Synchronization and Data Transformation, see Differences Between Data Synchronization and Data Transformation.

    Script

    It is used to process data by controlling the script syntax.

    SQL Script

    Used to write SQL statements to process the data in the database.

    Shell ScriptUsed to connect to external independent data processing processes such as Kettle tasks and Python calculation tasks by calling shell scripts.
    Python ScriptUsed to call Python scripts.
    Bat ScriptUsed to call batch files in remote Windows environments.

    Process

    It is used for complex logic processing.

    Parameter Assignment

    Used to output the upstream node results as parameters for use by downstream nodes.

    Conditional BranchUsed to determine whether to run the downstream node or which of the downstream nodes to run based on a condition from the upstream node or the system.
    Virtual NodeUsed to achieve the parallel running of multiple upstream nodes.
    Invocation TaskUsed to call other tasks in the current task and set the dependency between tasks to realize task orchestration.
    NotificationUsed to notify specified users of task scheduling results via WeCom group robot messages/email/DingTalk messages/SMS.
    Loop ContainerUsed for define loops for fetching data.

    Other

    It is used to add remarks for tasks.

    Remark

    Used to add remarks to a node or task, such as the task application scenario.

    The node usage explanation appears when you hover the cursor over a node in the node list, helping you quickly know how to use the node. See the following figure:

     2no.png

    Processes

     3.jpg

    Order
    DescriptionDocument

    1

    Configure the data source for data read and data write when designing a scheduled task.

    [General] Data Connection Configuration

    2

    Create and develop a scheduled task.

    See the document of each node.

    3

    Debug the task according to the task running log after manually executing the task.

    Running Log Introduction

    4

    Publish the task.

    /

    5

    Set a scheduling plan.

    Task Control of Scheduled Task

    6

    Manage scheduled tasks and resources.

    Scheduled Task O&M

    Load Distribution

    Notes

    SQL Statement Comment Description

    In versions before 4.0.17, the processing of SQL statement comments is ignored. In FineDataLink 4.0.17 and later versions, the SQL statement comments are created in the database.

    Ensure the format of SQL statement comments is correct: A space is required after a single-line comment. The Hive data source does not support multi-line comments.


    附件列表


    主题: Data Development - Scheduled Task
    • Helpful
    • Not helpful
    • Only read

    滑鼠選中內容,快速回饋問題

    滑鼠選中存在疑惑的內容,即可快速回饋問題,我們將會跟進處理。

    不再提示

    8s后關閉

    Get
    Help
    Online Support
    Professional technical support is provided to quickly help you solve problems.
    Online support is available from 9:00-12:00 and 13:30-17:30 on weekdays.
    Page Feedback
    You can provide suggestions and feedback for the current web page.
    Pre-Sales Consultation
    Business Consultation
    Business: international@fanruan.com
    Support: support@fanruan.com
    Page Feedback
    *Problem Type
    Cannot be empty
    Problem Description
    0/1000
    Cannot be empty

    Submitted successfully

    Network busy