4.1.4
/
4.2.1.4
Data Inspection supported the configuration of the concurrency quantity.
4.2.4.3
Merged Real-Time Task and Pipeline Task into Real-Time Module.
Since the execution of scheduled tasks, pipeline tasks, and APIs in Data Service in FineDataLink all require memory and concurrency resources, you may need to adjust the memory and concurrency allocated to them based on actual usage.
FineDataLink provides independent resource control for scheduled tasks, real-time tasks, pipeline tasks, and APIs in Data Service.
You are allowed to control resources in Load Distribution under System Management > Intelligent O&M > Load Management.
1. The execution of pipeline tasks, scheduled tasks, and real-time tasks is limited by both memory and concurrency. If either condition is not met, an error or prompt will be reported.
2. With Use permission on Intelligent O&M in System Management, you can use the Load Management function. For details, see Load Management Use Permission.
Scheduled Task
Memory limit
Specifies the maximum memory proportion of Real-Time Module.
Controls the maximum number of concurrent data synchronization tasks.
Concurrency limit
Specifies the maximum number of concurrencies.
Spark memory limit
Specifies the Spark memory proportion.
Affects computational complexity and speed.
Real-Time Module
Controls the memory and concurrency resources allocated to real-time tasks and pipeline tasks.
Specifies the maximum memory proportion, with the memory of real-time tasks defaulting to 256 MB.
For FineDataLink of V4.1.13.4 and later versions, you can adjust the memory limit by modifying FineDB configuration. For details about the modification method, you can contact Fanruan technical support by sending an email to international@fanruan.com or clicking at https://help.fanruan.com/finedatalink-en/.
Concurrency control
Specifies the maximum number of concurrencies, with the concurrency of real-time tasks defaulting to 4.
Data Service
Specifies the maximum memory proportion of the Data Service module.
Controls concurrency and the amount of data returned per request.
Data Inspection
Controls the number of concurrent tasks.
For FineDataLink of V4.1.4 and later versions, changes in the memory limit are as follows:
If the memory limit is set in independently deployed FineDataLink projects before the upgrade, this configuration will become invalid after the upgrade. If you need to modify the memory limit again, you can contact Fanruan technical support.
For FineDataLink versions earlier than V4.1.4, if you modify the number of concurrent tasks, the previous concurrency settings will become invalid after the upgrade. You need to reconfigure the settings in Concurrency Control on the Load Distribution tab page.
You can adjust the memory proportion of each module in the Memory Distribution area.
A
Displays the total memory in a stand-alone environment.
Displays the memory available for Data Development, Data Pipeline, and Data Service.
B
Displays the memory that can be allocated to scheduled tasks.
C
Displays the memory that can be allocated to real-time tasks and pipeline tasks.
D
Displays the memory that can be allocated to APIs in Data Service.
E
Displays the public space memory, which is represented by the green bar. If the memory used by a single module exceeds the configured limit, the module can compete for available memory from the public space memory.
F
The configurable memory proportion ranges from 10% to 60% of the total memory.
The memory reserved for the system and components is 40% of the total memory, which defines the upper limit of the configurable memory.
G
1. In a cluster environment, the available memory for all nodes is displayed.
2. In the current version, the default upper limit for the estimated memory of Data Pipeline is 1 GB, and any estimation exceeding 1 GB is still calculated as 1 GB.
You can click the Edit button in the upper right corner to configure the memory proportion by drag-and-drop operations, as shown in the following figure.
For example, if you encounter the following error during the execution of a scheduled task, it indicates that the memory limit is exceeded.
In this case, you need to increase the memory proportion for the Scheduled Task module, as shown in the following figure.
If you are prompted that the memory limit is exceeded during the startup of a pipeline task, as shown in the following figure:
You can configure Memory Distribution to increase the memory proportion for Real-Time Module.
You can adjust the number of concurrent tasks for Scheduled Task, Real-Time Module, and Data Inspection in the Concurrency Control area.
Displays the number of CPU cores and the total number of configurable concurrencies in FineDataLink.
In a cluster environment, the maximum number of concurrencies for all nodes is displayed. For example, the maximum number of concurrencies for Node 1 is Y, and the maximum number of concurrencies for Node 2 is Y.
You can adjust the concurrency proportion by drag-and-drop operations, with 0.5 as the minimum adjustment unit.
The adjustable range of the maximum number of concurrencies:
0.5 * the number of CPU cores ≤ the concurrent number of pipeline tasks and scheduled tasks ≤ 9.5 * the number of CPU cores
In a cluster environment, the number of CPU cores of each node is displayed, but all nodes must be uniformly configured.
The Number of Concurrencies in Data Development:
The actual number of concurrencies in the Data Synchronization/Data Transformation/Parameter Assignment nodes is limited by the smallest value between the thread pool and concurrency proportion set in Concurrency Control.
The thread pool of Data Synchronization/Data Transformation: Defaults to 1 * the total number of concurrent data development tasks.
The thread pool of Shell Script/Bat Script: Defualts to 1 * the total number of concurrent data development tasks.
The thread pool of SQL Script: Defaults to 1 * the total number of concurrent data development tasks.
Other nodes in scheduled tasks (including SQL Script, Shell Script, Bat Script, and Python Script) are not limited by Concurrency Control. They are only limited by their own thread pool.
The number of concurrencies in Real-Time Module includes the number of concurrent pipeline tasks and real-time tasks.
The number of concurrent tasks in Data Pipeline: The thread pool of pipeline tasks occupies 0.25 * the total number of concurrent pipeline tasks, with a minimum of 8. One pipeline task requires 4 concurrencies.
The available number of concurrencies is calculated as the configured number of concurrencies - the number of concurrencies occupied by the thread pool of pipeline tasks.
For example, if the total number of threads for pipeline tasks is 30, of which 8 is for data writing by default, then 22 threads remain. Since one pipeline task requires 4 concurrencies, you can still configure 5 pipeline tasks.
The number of concurrent real-time tasks: The execution of a real-time task will start after the system detects 4 concurrencies available for the task, so you need to reserve sufficient execution resources in Load Distribution.
The number of concurrencies in Data Inspection: Defaults is 0 and requires manual adjustment. One data inspection task occupies one concurrency.
After clicking the Edit button in the upper right corner, you can configure the maximum concurrency by drag-and-drop operations, as shown in the following figure.
For example, during the execution of a pipeline task, if the concurrency limit is exceeded, a prompt will be displayed at the startup of the task, as shown in the following figure.
In this case, you can increase the number of concurrencies in Real-Time Module by configuring Concurrency Control, as shown in the following figure.
Note: If the concurrency limit is exceeded during the execution of a scheduled task, a message will be recorded in the log, as shown in the following figure.
滑鼠選中內容，快速回饋問題
滑鼠選中存在疑惑的內容，即可快速回饋問題，我們將會跟進處理。
不再提示
10s後關閉
Submitted successfully
Network busy