FDL Deployment Environment Preparation

Note: The above operating systems are recommended.  You can contact FanRuan technical support if you encounter problems deploying FineDataLink on other Linux systems.

The project can be deployed in both wide area networks and local area networks. The requirements for the network environment are shown in the following table.

Bandwidth is mainly used to limit the amount of data transmitted over a network connection in a given amount of time, which can be calculated using the following formula. 

Note:

Note:

 

Bandwidth (Mb/s) * 80%/8 (bit) = Traffic (Mb/s). 

Multiplying the bandwidth by 80% is to reserve a bandwidth margin to avoid network congestion in the actual transmission.

You can refer to the following table.

After conducting numerous internal tests using tens of millions of data records, it has been found that the transmission traffic is close to a value corresponding to the bandwidth of 50 Mb/s (not exceeding 100 Mb/s). Therefore, bandwidth between 50 Mb/s and 100 Mb/s is recommended.

Resource Control Memory 

Note: For details of resource control settings, see Load Distribution.

Unknown Number of Scheduled Tasks and Pipeline Tasks (Applicable to Newly Deployed Projects)

Known Number of Scheduled Tasks and Pipeline Tasks (Applicable to Project Migration and Upgrade)

Minimum memory size = MAX(minimum memory size for running scheduled tasks, minimum memory size for running pipeline tasks)

1. For 4.1.55 and later releases:

Take the following task an an example, the required memory is described below.

Input nodes (three): (8 + 1 * 2 * 24) * 3

Process node (one): 64 + 2 * 2 * 64

Output nodes (two): 32 + 32

Total: 552 MB

You can find the description in the corresponding log.

2. For releases before 4.1.55:

a. Estimated memory required for running scheduled/pipeline tasks (applicable to multi-task scenarios where accurate calculation is impossible)

b. Accurate memory required for running scheduled/pipeline tasks

The following is an example of calculating the memory of a scheduled task.

Input: 64 MB + Output node quantity * (64 MB +64 MB)

Process:  64 MB + Ultimate output node quantity * (64 MB +64 MB)

Output: Output node quantity * 32 MB

Calculate the memory used by the task:

Input: 2 * (64 MB + 1 * (64 MB +64 MB)) = 384 MB

Process:  64 MB + 3 * (64 MB +64 MB) = 448 MB

Output: 3* 32 MB =96 MB

Total: 928 MB

Web Container Memory 

 The Web container memory should be equal to or larger than the resource control memory. For example, if an initialized project has a resource control memory of 16 GB, the Web container memory should be set to a value higher than 16 GB, but not more than 80% of the system memory (recommended).

Note 1: For details of modifying the container memory, see Tomcat Memory Modification.

Note 2: For details of the resource control memory, see the Resource Control Memory section of this article.

System Memory 

The system memory should be equal to or greater than the Web container memory (with a recommended value of less than 80% of the system memory).

Note: The larger the system memory, the stronger the system scalability during use. For example, to increase the number of concurrent computations from four to eight, you just need to increase the resource control memory and the Web container memory.

CPU Configuration 

The number of threads should be at least twice the number of concurrent tasks.

To ensure the high performance of concurrent transmission, the number of CPU threads can be slightly greater than twice the number of concurrent tasks.

The CPU mainly limits the number of scheduled tasks and pipeline tasks that run concurrently.

The number of CPU threads affects full-volume synchronization of the data pipeline task, whereas incremental synchronization remains unaffected.

Disk Space

The disk space shall be more than 50 GB.

Disk space is mainly occupied by files (installation files, task files, log files, and backup files) and task read and write throughput. Data table read and write mainly uses the memory, thus requiring little disk space if the memory is sufficient.

In summary, reserve at least 50 GB of disk space for server deployment. You can increase the disk space as needed if you need to store Excel and CSV files in the server's local directory.

Contact the technical support personnel for the installation package.

1. For details about port occupation, see Viewing Port Usage.

2. If the default port number conflicts with that of other projects, modify the port number and then open the corresponding port.

3. To deploy multiple Tomcat projects on a server, modify the Tomcat port number to prevent port conflict. For details, see Modifying Tomcat Port Number.

4. If the firewall is enabled, you need to open the relevant port. For a Windows system, see Setting Inbound and Outbound Rules for Windows Server. For a Linux system, see Using and Configuring Linux Firewall.

5. For environments with strict port restrictions between Docker containers or servers, open ports between the node servers for inter-node communication.

• If you use the TCP protocol, open the following ports: 7800, 7810, 7820, 7830, 7840, 7850, 7860, and 7870.

• If you use the UDP protocol, the port number for inter-node communication is a random one between 45588 and 65536.