Description of Scheduled Task Capabilities

The Scheduled Task module in Data Development supports data extraction, transformation, and loading using various nodes and operators on a visual page. It enables the automatic execution of scheduled tasks through the Timed Scheduling function, helping you build offline data warehouses effortlessly and ensuring efficient and stable data production.

This article describes the capabilities of the Scheduled Task module.

Flexible Selection of Source and Target Terminals 

Function Description

For details, see the "Types of Data Sources Supported by Scheduled Task" section of Data Sources Supported by FineDataLink.

The data sources in the first and second columns in the following table can be combined flexibly. For example, you can read file data and write the data into a database after processing (optional).

Others

Flexible Conversion of Data Structures 

Scheduled Task enables you to convert data to structured data, semi-structured data, and unstructured data flexibly.

Example:

Data Cleaning

Clean the raw data to output standardized data.

Data Update

The data update methods include incremental update, full update, and comparison-based update. For details, see Overview of Data Synchronization Method.

Note:

You can use the Data Comparison operator to check the data consistency by comparing source and target data.

Data Integration

Integrating various types of heterogeneous data is supported to achieve data association and dimensional data modeling. For details, see Union All, Data Association, SQL Script, and Spark SQL.

Data Alert

Scheduled Task provides instant alerts for key business data when limits are reached. For details, see Notification.

Notification of scheduled task execution failure is also supported. For details, see Task Control - Result Notification.

Data Sharing

Note:

You can use Data Service to output data as APIs for other systems to call.

Data can also be output as files for storage. For details, see Function Description of File Output Operator and File Transfer. 

Others

1. You can be prompted for DDL changes in the source table of a scheduled task and notify specified users. For details, see Synchronizing DDL Changes Using Scheduled Task.

2. Data Lineage helps you understand the entire data production process and the impact scope of modifications to data tables. For details, see Lineage Analysis.

3. The front-end Database Table Management function enables visual management of multi-source data and helps understand data conditions quickly by searching for specific database tables.

Task development and editing are separated from task execution. You can edit and modify scheduled tasks in a development environment, where you can also perform task commissioning, and then publish them to a stable production environment, ensuring that task development and execution do not interfere with each other.

For details, see Development Mode and Production Mode. 

The canvas-like development page enables efficient task development and easy O&M.

For details about differences between Data Synchronization and Data Transformation, see Differences Between Data Synchronization and Data Transformation.

Combining Various Scheduling Strategies Flexibly

FineDataLink provides three scheduling strategies, namely time-based, event dependency-based, and trigger-based scheduling, which can be flexibly combined to meet diverse scheduling needs.

For details, see Overview of Scheduling Plan.

Setting Priority for a Single Scheduled Task

You can set task priority based on the business data priority and set the log level to ensure focused support for critical businesses.

For details, see Task Control - Task Attribute.

Ensuring Normal Task Execution with Fault Tolerance Mechanisms

FineDataLink offers fault tolerance mechanisms, including Timeout Limit, Retry After Failure, and Dirty Data Tolerance. When encountering controllable risks such as network fluctuations or a small amount of dirty data, the platform automatically retries the task to ensure smooth task operation.

For details, see Task Control - Fault Tolerance Mechanism.

Notification of Task Execution Failure

With the result notification enabled, the platform will notify the relevant personnel in case of task execution failure.

• The notification channel includes platform messages, emails, SMSs, DingTalk chatbot, Lark chatbot, and WeCom chatbot.

• The notification content includes task exceptions, dirty data, and notification of source table structure changes.

For details, see Task Control - Result Notification.

Recycle Bin

You can restore and hard delete the deleted tasks in Scheduled Task, Data Pipeline, and Data Service in Recycle Bin.

For details, see Recycle Bin.

1. You can monitor and manage scheduled tasks in real time under O&M Center > Scheduled Task > Running Record, where you can view task running information such as the running status, duration, and triggering method, and retry scheduled tasks (for supplementing data or re-executing tasks after solving the dirty data that causes data writing failure).

For details, see Running Record.

2. You can set the execution frequency and the event-based scheduling plan for multiple/single scheduled tasks and modify and delete the execution frequency and the event-based scheduling plan of the task under O&M Center > Scheduled Task > Scheduling Plan.

For details, see Scheduled Task O&M - Scheduling Plan. 

3. You can view task information such as the number of scheduled tasks, task scheduling status, and scheduling plan execution status, and configure result notification for tasks in batches under O&M Center > Scheduled Task > Task Management.

For details, see Scheduled Task O&M - Task Management.

4. You can manage the memory resources used by scheduled tasks for an independently deployed FineDataLink project.

For details, see Load Distribution.

Permission related to the Scheduled Task module is explained in the following table.