You can deploy FineDataLink projects via FineOps.
This document provides multiple standard schemes based on the number of scheduled tasks, real-time tasks, and pipeline tasks. Choose the one that best meets your needs.
Number of scheduled tasks: Estimate the number of data tasks to be developed.
Number of real-time pipeline tasks: Confirm the number of tablespaces in the business database. One tablespace corresponds to one pipeline task. This refers only to real-time pipelines.
Number of real-time tasks: Estimate the number of database tablespaces whose data needs to be synchronized in real time.
The schemes in this document are designed for new project deployment scenarios, providing recommended server configuration based on business usage. You can select the scheme according to the task quantity.
The schemes in this document are intended for scenarios where the volume of data (calculated by Row count * Column count) that changes within one hour in a single business database does not exceed one million. If the data change volume surpasses this limit, contact FanRuan technical support personnel for further assistance.
These schemes offer resource recommendations based on typical scales of customers’ projects. If the number of tasks exceeds the applicable range, contact FanRuan technical support personnel for further assistance.
This document provides recommended configuration for newly deploying all FineDataLink components. If you have prepared some components such as the configuration database and the file service, you can lower the corresponding server configuration. For details, see the notes of each scheme.
As usage increases over time, if your current server resources become insufficient and require expansion, contact FanRuan technical support personnel for assistance.
In the schemes in this document, FineOps and the FanRuan project are deployed on the same server, so disaster recovery cannot be guaranteed. If the server fails, FineOps and the FanRuan project become unavailable at the same time. FineOps cannot quickly restore the FanRuan project from offsite backups, which prevents disaster recovery from working as designed. If disaster recovery is required and more server resources are available, deploy FineOps, the main applications of the FanRuan project, and the project components on separate servers to isolate failures and enable quick offsite recovery. For details about split deployment, contact FanRuan technical support.
The number of scheduled tasks is less than 35.
Pipeline tasks are not needed.
It is designed for customers with minimal business usage only, where FineOps Basic Edition instead of FineOps Standard Edition should be used. Monitoring indicators in Tracing and Health Inspection are unavailable.
Server: 8-core CPU, 16 GB of memory, and 300 GB of disk space
It is used to deploy one FineDataLink component, one Nginx component (an internal gateway), and one MySQL component (a configuration database).
It is used to deploy all FineOps components excluding Elasticsearch and SkyWalking OAP.
The number of scheduled tasks is less than 150.
Server: 8-core CPU, 32 GB of memory, and 500 GB of disk space
It is used to deploy one FineDataLink component, one Nginx component, and one MySQL component.
It is used to deploy all FineOps components.
The number of scheduled tasks is less than 300.
The number of real-time pipeline tasks is less than 16.
The number of real-time tasks is less than 20.
Server: 16-core CPU, 64 GB of memory, and 2 TB of disk space
It is used to deploy one FineDataLink component, one Kafka component, one Nginx component, and one MySQL component.
The number of scheduled tasks is less than 500.
The number of real-time tasks is less than 40.
1. FineDataLink main application server: 16-core CPU, 64 GB of memory, and 500 GB of disk space
It is used to deploy one FineDataLink component.
2. Server for FineDataLink components and FineOps: 8-core CPU, 32 GB of memory, and 2 TB of disk space
It is used to deploy one Kafka component, one Nginx component, and one MySQL component.
The number of scheduled tasks is less than 500 * N.
The number of real-time pipeline tasks is less than 16 * N.
The number of real-time tasks is less than 40 * N.
1. The number of FineDataLink main application servers is N.
1. Each main application server: 16-core CPU, 64 GB of memory, and 500 GB of disk space
Each one should be deployed with one FineDataLink component.
It is used to deploy one Kafka component, one Nacos component, one Nginx component, and one MySQL component.
Note:
1. This scheme is intended for customers with little business usage and limited server resources only.
This scheme is only applicable to deploying FineOps Basic Edition instead of FineOps Standard Edition. Monitoring indicators in Tracing and Health Inspection are unavailable.
2. In the schemes in this document, FineOps and the FanRuan project are deployed on the same server, so disaster recovery cannot be guaranteed. If the server fails, FineOps and the FanRuan project become unavailable at the same time. FineOps cannot quickly restore the FanRuan project from offsite backups, which prevents disaster recovery from working as designed.
If disaster recovery is required and more server resources are available, deploy FineOps, the main applications of the FanRuan project, and the project components on separate servers to isolate failures and enable quick offsite recovery. For details about split deployment, contact FanRuan technical support.
It requires one server with 8 CPU cores, 16 GB of memory, and 300 GB of disk space.
It is used to deploy one FineDataLink component, one Nginx component, one MySQL component, and Elasticsearch log storage.
Component to be installed
It is used to install all components of FineDataLink and FineOps.
1. For details about the role of FineDataLink components, see Understanding the Roles of O&M Project Components.
FineDataLink
Nginx
MySQL (The self-provided component is supported.)
Elasticsearch (The self-provided component is supported.)
Filebeat
OPS Agent
2. For details about the role of FineOps components, see Understanding the Roles of FineOps Components.
OPS, Nginx, and OPS Agent
Pushgateway, Prometheus, Grafana, and Alertmanager
Registry (The self-provided component is supported.)
Supported:
Ubuntu 18.04.4 and later releases (Ubuntu 20.04 is not supported.)
CentOS 7.3 to 7.9
Red Hat 7.6 and later releases
Rocky Linux 8.8 to 9.4
1. Since CentOS is discontinued, you are advised to use Ubuntu.
2. When using Ubuntu, verify the user privileges (as the default root user is not a super admin).
CPU clock speed
Remaining free physical memory
16 GB
Registration and authentication must be completed after project deployment.
For an external network or an internal network that can connect to the FanRuan cloud, you are advised to authenticate registration via a public cloud. For details, see Public Cloud Authentication.
For the pure internal environment, you need to install a registration service component on the project component server and authenticate registration via private cloud on FineOps. Therefore, you must ensure at least 2 GB of additional available physical memory on this server. For details, see New Project Registration.
The server shall have a partition with remaining free space of more than 300 GB.
Note
:
For details, see Preparing the FineDataLink Mounting Directory.
For details, see Preparing the FineOps Server.
Otherwise, containers may fail to access these directories, leading to data loss or application startup issues.
Sharing the file system may cause performance degradation, file permission issues, and data inconsistency, affecting the reliability and response speed of applications running in the container.
Ensure the server has the tar command installed.
The tar command is a commonly used tool for packaging and compressing files.
FineOps requires this command for file extraction.
Ensure the server has the sed command installed.
The sed command is used for text processing.
FineOps requires this command for text processing.
Ensure the user you use can connect to the FineOps server via the SSH protocol.
Ensure the password used for SSH connection contains no English single quotation marks ('), or the privilege will fail to be validated during deployment.
1. Using the root user (superuser) for project deployment and operation is preferred.
2. To use a non-root user for deployment and operation, see Linux User Privilege Explanation.
At least 100 Mbps (12.5 MB/s)
At least 10 Mbps (1.25 MB/s)
Component-used port and component port interconnection
For details, see Confirming Server Network of the FineDataLink Project.
If the server is running the Kylin operating system, ensure that Podman is not installed on the server.
The version of runc bundled with Podman is too low and conflicts with the Docker installation managed by FanRuan, which may cause deployment failures.
Refer to the relevant document to uninstall Podman. You are advised to verify the impact of uninstallation in a non-production environment first.
Ensure that Kingsoft Antivirus is not installed on the server.
This antivirus software may trigger server connection interruptions when the dockerd command is executed, resulting in deployment failures.
Refer to the relevant document to uninstall Kingsoft Antivirus. You are advised to verify the impact of uninstallation in a non-production environment first.
In the schemes in this document, FineOps and the FanRuan project are deployed on the same server, so disaster recovery cannot be guaranteed. If the server fails, FineOps and the FanRuan project become unavailable at the same time. FineOps cannot quickly restore the FanRuan project from offsite backups, which prevents disaster recovery from working as designed.
It requires one server with 8 CPU cores, 32 GB of memory, and 500 GB of disk space.
Elasticsearch and SkyWalking OAP
32 GB
The server shall have a partition with remaining free space of more than 500 GB.
1. There must be a single partition that meets the condition.
2. The required disk space decreases if you use self-prepared components.
If you use a self-provided Elasticsearch component, the required remaining free disk space on the server can be reduced by 100 GB.
If you use a self-provided external configuration database component, the required remaining free disk space on the server can be reduced by 50 GB.
At least 1000 Mbps (125 MB/s)
It requires one server with 16 CPU cores, 64 GB of memory, and 2 TB of disk space.
It is used to deploy one FineDataLink component, one Kafka component, one Nginx component, one MySQL component, and Elasticsearch log storage.
Kafka (Self-prepared Kafka cannot be connected during project deployment, but can be connected after deployment.)
64 GB
The server shall have a partition with remaining free space of more than 2 TB.
If you use a self-provided Kafka component, the required remaining free disk space on the server can be reduced by 1.5 TB.
1. FineDataLink main application server: 16 CPU cores, 64 GB of memory, and 500 GB of disk space
2. Server for FineDataLink components and FineOps: 8 CPU cores, 32 GB of memory, and 2 TB of disk space
It is used to deploy one Kafka component, one Nginx component, one MySQL component, and Elasticsearch log storage.
It is used to install FineDataLink application components.
1. It is used to install the pipeline and the O&M components of FineDataLink.
The time on all servers must be synchronized, with a maximum difference of no more than 5 seconds.
Inconsistent server time can cause issues such as incorrect execution of scheduled tasks, disorganized log records, and data inconsistencies.
Inconsistent server time zones can cause issues such as incorrect execution of scheduled tasks, disorganized log records, and data inconsistencies.
There must be a single partition that meets the condition.
Each main application server: 16 CPU cores, 64 GB of memory, and 500 GB of disk space
It is used to deploy one Kafka component, one Nacos component, one Nginx component, one MySQL component, Elasticsearch log storage, and cluster components.
For details about the role of FineDataLink components, see Understanding the Roles of O&M Project Components.
1. It is used to install the pipeline, cluster, and the O&M components of FineDataLink.
Nacos and Nacos MySQL
Elasticsearch, MinIO, MySQL, and Redis (The self-provided component is supported.)
If you use a self-provided file storage component, the required remaining free disk space on the server can be reduced by 100 GB.
At least 1000 Mbp/s (125 MB/s)
The configuration requirements in the above sections must be met because each component of FineDataLink projects has minimum operational requirements, as shown in the following table.
1. To prevent resource over-provisioning, a CPU-sharing strategy is used, as the components do not run at full load simultaneously. Therefore, the required number of server CPU cores is not the total number of maximum CPU cores used by each container.
Note 2. The server memory size must be at least the total of the maximum memory used by each container, to implement the Java memory configuration for each container, including Xmx, off-heap memory, FineIO read/write memory, CodeCache, and Metaspace.
/
Default: Server memory - 8 GB
30 GB at most
滑鼠選中內容,快速回饋問題
滑鼠選中存在疑惑的內容,即可快速回饋問題,我們將會跟進處理。
不再提示
10s後關閉
Submitted successfully
Network busy