Operating System Requirement
Category | Model | |
|---|---|---|
Linux | CentOS 6 | CentOS 6.5, CentOS 6.6, CentOS 6.7, CentOS 6.8, and CentOS 6.9 |
CentOS 7 | CentOS 7.0, CentOS 7.1, CentOS 7.2, CentOS 7.3, CentOS 7.4, CentOS 7.6, CentOS 7.8, and CentOS 7.9 | |
Red Hat 6 | Red Hat 6.5, Red Hat 6.6, Red Hat 6.7, Red Hat 6.8, and Red Hat 6.9 | |
Red Hat 7 | Red Hat 7.0, Red Hat 7.1, Red Hat 7.2, Red Hat 7.3, Red Hat 7.4, Red Hat 7.7, Red Hat 7.9, and Red Hat 8.0 | |
| Ubuntu | Ubuntu 10.04.4, Ubuntu 18.04, and Ubuntu 20 | |
| Amazon Linux | Amazon Linux AMI 2018.03 | |
| Rocky Linux | Rocky Linux 8.8 to 9.4 | |
Windows | Windows Server 2008 and later releases and Windows 11 | |
Configuration database | Amazon RDS for MySQL, MySQL, SQL Server, Oracle, Db2, and PostgreSQL  Note: | |
Browser | Chromium-based browser You are advised to use modern mainstream browsers (such as Google Chrome and Microsoft Edge) of the latest versions. | |
| Google Chrome | The browser version should be later than 94 for FineDataLink 4.1.x to 4.1.6.x. The browser version should be later than 94 for FineDataLink of versions later than 4.1.6.x. You are advised to use browsers of the latest versions. | |
| Microsoft Edge | ||
Note that:
1. The above operating systems are recommended for use. To deploy FineDataLink on other Linux systems, contact FanRuan technical support for assistance by sending an email to international@fanruan.com.
2. You need to deploy Kafka to use Data Pipeline, which is preferred to be installed on a Linux system. (Kafka installed on a Windows system is not suitable for production environments and can be used for demonstration purposes only, as it has limited performance.) You can deploy Kafka and FineDataLink on different servers.
Server Configuration
Configuration of CPU, Memory, Disk, and Network
This document provides multiple standard schemes based on the number of scheduled tasks, real-time tasks, and pipeline tasks. Choose the one that best meets your needs.
Scheme Applicability
The schemes in this document are designed for new project deployment scenarios, providing recommended server configuration based on business usage. You can select the scheme according to the task quantity.
The schemes in this document are intended for scenarios where the volume of data (calculated by Row count * Column count) changed in an hour in a single business database does not exceed one million. If the data change volume surpasses this limit, contact FanRuan technical support personnel for further assistance.
These schemes offer resource recommendations based on typical scales of customers’ projects. If the number of tasks exceeds the applicable range, contact FanRuan technical support personnel for further assistance.
As usage increases over time, if your current server resources become insufficient and require expansion, contact FanRuan technical support personnel for assistance.
Scheme Introduction
| Scheme | Applicable Condition | Server Configuration |
|---|---|---|
| Minimum Requirement | The number of development tasks is less than 35. Pipeline tasks are not needed. |  Note: 1. Server: 8-core CPU, 16 GB of memory, and 300 GB of disk space 2. Network and port:
|
| Scheme One | The number of scheduled tasks is less than 150. Pipeline tasks are not needed. | 1. Server: 8-core CPU, 32 GB of memory, and 300 GB of disk space 2. Network and port:
|
| Scheme Two | The number of scheduled tasks is less than 300. The number of pipeline tasks is less than 16. The number of real-time tasks is less than 20. | 1. Server: 16-core CPU, 64 GB of memory, and 300 GB of disk space 2. Network and port:
|
| Scheme Three | The number of scheduled tasks is less than 500. The number of pipeline tasks is less than 16. The number of real-time tasks is less than 40. | 1. Server: 16-core CPU, 64 GB of memory, and 500 GB of disk space 2. Network and port:
|
| Scheme Four | The number of scheduled tasks is less than 500 * N. The number of pipeline tasks is less than 16 * N. The number of real-time tasks is less than 40 * N. | 1. The number of FineDataLink main application servers is N. Each main application server: 16-core CPU, 64 GB of memory, and 500 GB of disk space Each one should be deployed with one FineDataLink component. 2. Network and port:
|
Port Preparation
This section describes the ports to be opened for FineDataLink deployment.
Port Opening Notes
1. If the default port number conflicts with that of other projects, modify the port number and then open the corresponding port.
2. If you want to deploy multiple Tomcat projects on a server, modify the Tomcat port number to prevent port conflict.
3. If the firewall is enabled, you need to open the relevant port.
4. For the Docker container or environments with strict port restrictions between servers, it is necessary to open ports between the servers of nodes for inter-node communication.
If you use the TCP protocol, open ports 7800, 7810, 7820, 7830, 7840, 7850, 7860, and 7870.
If you use the UDP protocol, open ports ranging from 45588 to 65536 since random ones will be used.
Port Description
| Classification | Content | Port Number | Remark |
|---|---|---|---|
| Standalone | A project-occupied port | 8068 | It is an external port that can be closed to the public and can be modified (in server.xml). You can configure SSL and a short address in the load balancer. |
A port for the crash-handling tool | 12100 | / | |
| A port for the gpfdist service | 15500 | / | |
| Cluster | A project-occupied port | 8068 | It is an external port that can be closed to the public and can be modified (in server.xml). You can configure SSL and a short address in the load balancer. |
| A port for the crash-handling tool | 12100 | / | |
| A port for TCP communication in the cluster | 7800 7830 7840 7850 7870 | / | |
| A port for the gpfdist service | 15500 | / | |
| Notification | WebSocket port | Starting from FineDataLink 4.0.6, the default port number ranges from 58888 to 59888. | / |
| WebSocket forwarding port | Starting from FineDataLink 4.0.6, the default port number is 58889. |
Resource Control Memory
Note:
Unknown Number of Scheduled Tasks and Pipeline Tasks (Applicable to Newly Deployed Projects)
For details, see the "Configuration of CPU, Memory, Disk, and Network" section.
Known Number of Scheduled Tasks and Pipeline Tasks (Applicable to Project Migration and Configuration Optimization Scenarios)
The minimum memory size shall be the greater one between the minimum memory size for running scheduled tasks and for running pipeline tasks.
| Accuracy | Memory |
|---|---|
Accurate | The minimum memory size shall be the greater one between the minimum memory size for running scheduled tasks and for running pipeline tasks. For details, see the following table. |
Rough | The minimum memory size shall be the greater one between the minimum memory size for running scheduled tasks and for running pipeline tasks. Assume that each scheduled or pipeline task requires 1 GB of memory. |
1. For 4.1.55 and later releases:
Type | Node | Memory | ||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Scheduled task | Single input node | Calculation formula: Buffer size + outputSize * 2 * Channel size The formula is described below.
For inputs from non-relational DB tables (such as Jodoo and Mongo) and other inputs (such as API input and file input), a Reader takes up 64 MB of memory. For inputs from relational DB tables, the size depends on table structures. Allocate 1 MB of memory for each column of the input table, and in particular, 2 MB if the precision of the column exceeds 1024. The resulting size shall be a positive multiple of 8 MB and not exceed 64 MB. For example, if a table has the following structure, it takes 9 MB of memory and shall be allocated with 16 GB of memory. For a table of the following structure:
The calculation of channel memory is relatively complex. Generally, it takes 8 MB or 16 MB of memory, not exceeding 64 MB.
It is the number of succeeding nodes connected with the input node. | ||||||||||||||||||
Process node | 64 + outputSize * 2 * 64
It is the sum of the number of output nodes and Python nodes that are directly connected with the process node. The succeeding process node of the process node is not included. | |||||||||||||||||||
Single output node | 32 MB An output node usually takes 32 MB of memory. Specifically, if the data is output to a Doris or StarRocks database, a single output node takes 90 MB of memory. | |||||||||||||||||||
Pipeline task | / | The memory calculation method of input and output nodes is the same as that of the scheduled task. |
Take the following task as an example, the required memory is described below.
Input nodes (three): (8 + 1 * 2 * 24) * 3
Process node (one): 64 + 2 * 2 * 64
Output nodes (two): 32 + 32
Total: 552 MB
You can find the description in the corresponding log.
2. For FineDataLink of versions before 4.1.55:
Estimated memory required for running scheduled/pipeline tasks (applicable to multi-task scenarios where accurate calculation is impossible)
| Type | Node | Memory |
|---|---|---|
| Scheduled task | / | 1024 MB per task |
| Pipeline task | / | 1024 MB per task |
Accurate memory required for running scheduled/pipeline tasks
| Type | Node | Memory |
|---|---|---|
| Scheduled task | Single input node | 64 MB + 128 MB * Number of output channels |
| All process nodes | 64 MB + 128 MB * Number of connected output nodes | |
| Single output node | 32 MB | |
| Pipeline task | / | 1024 MB per task |
The following is an example of calculating the memory required for a scheduled task.
Input: 64 MB + Output node quantity * (64 MB +64 MB)
Process: 64 MB + Ultimate output node quantity * (64 MB + 64 MB)
Output: Output node quantity * 32 MB
Calculate the memory used by the task:
Input: 2 * (64 MB + 1 * (64 MB + 64 MB)) = 384 MB
Process: 64 MB + 3 * (64 MB + 64 MB) = 448 MB
Output: 3 * 32 MB =96 MB
Total: 928 MB
Web Container Memory
The Web container memory should be equal to or larger than the resource control memory. For example, if a new project has a resource control memory of 16 GB, the Web container memory should be set to a value higher than 16 GB, but not more than 80% of the system memory (recommended).
Deployment Package Preparation
Contact the technical support personnel for the installation package. To contact our technical support personnel, you can send an email to international@fanruan.com or click the icon in the lower right corner at https://help.fanruan.com/finedatalink-en/.
Previous：Version Compatibility Instruction
