Server Configuration Confirmation- FineOps Help Document

Last update: 2024-07-10

Overview

FineOps supports the deployment of FineBI 6.1 projects.

This article provides recommended server configurations for scenarios mainly involving extracted data. If the FineBI project uses direct-connected data only, you can reduce the memory of the worker component properly, exchange the memory configurations of the bi-web and the worker components, and keep the other configurations unchanged.

Ensure that the server meets the minimum configuration requirements, and then choose the appropriate deployment plan and the memory and CPU configurations for nodes according to the data volume and the number of daily active users of the project.

The nodes are described in the following table.

Node	Description
BI	BI business node
Master	Engine metadata node
Worker	Engine computing node
MinIO	Data storage node

Minimum Configuration

Basic Configuration

Configuration

Lowest Requirement

Operating System Type

Linux

Operating System Architecture

X86_64

Operating System Kernel

Version 3.10 and above

Operating System Software

Recommended: Ubuntu 22

Supported:

Ubuntu 18.04.4 and later releases (Ubuntu 20.04 is not supported.)
CentOS 7.3 to 7.9
Red Hat 7.6 and later releases

Basic Configuration

1. Prepare two new servers for deploying FineBI projects.

Prepare an additional server for deploying FineOps, which is listed in the table for reference and not part of FineBI resources.

You can use the df-h command to query the disk space. There must be a partition that meets the condition.

Server Type

CPU

Remaining Free Physical Memory

Remaining Free Disk Space

Deployment Content

Main application server

16 cores

64 GB

500 GB

One bi-web component, one worker component, one master component, and one bi-minio component

300 GB of disk space is enough if you don't deploy the bi-minio (data storage) component.

Component server

4 cores

16 GB

50 GB

The nginx component, the configuration database, the elasticsearch component, and the cluster component

FineOps server

4 cores

16 GB

120 GB

The ops component

2. Prepare two servers if there are insufficient resources and deploy both the ops component and some FineBI components on the component server.

You can use the df-h command to query the disk space. There must be a partition that meets the condition.

Server Type

CPU

Available Physical Memory

Available Disk Space

Deployment Content

Main application server

16 cores

64 GB

500 GB

One bi-web component, one worker component, one master component, and one bi-minio component

300 GB of disk space is enough if you don't deploy the bi-minio (data storage) component.

Component server

8 cores

32 GB

150 GB

The nginx component, the configuration database, the elasticsearch component, the cluster component, and the ops component

CPU Clock Speed

Minimum 2.5 GHz

Disk Type

For data storage components and clustered file service components, the disk type must be XFS to use the FineOps-deployed minio component. A non-XFS disk might result in data write failure even with available disk space.

Note:

If an Alibaba Cloud server is used, the default disk type is not XFS. Modify it.

This operation may involve disk formatting. FanRuan does not provide related guidance. Proceed cautiously under the guidance of your server vendor.

Disk Performance

The minimum sustained transfer rate must be at least 400 MB/s, meaning the disk can transfer a minimum of 400 MB of data per second.

The random IOPS (4K) must be greater than 20,000, meaning the disk can perform at least 20,000 read/write operations on 4 KB data blocks per second.

The sequential IOPS (1024K) must be greater than 315.5 MB, meaning the disk can transfer at least 315.5 MB of blocks per second.

Intranet Latency

Less than 1 ms

Server Bandwidth

Greater than 2.5 Gbps (312.5 MB/s)

User Permission

You need to provide the server username and password for project deployment and operation. Check in advance whether the server user's permissions meet the requirements.

You are advised to use the root user account for project deployment and operation.
To use a non-root user account for project deployment and operation, ensure this account has at least permissions for the following commands:

Permission Name	Application Scenario
sh	Used to execute multiple commands requiring sudo permissions at once.
echo sed cp sysctl	Used to modify some configuration files and kernel parameters, mainly to modify the sysctl.conf file in the /etc directory. If the parameter values are already set to target values, executing corresponding commands will not modify the values but will make these parameters effective. The parameters include vm.max_map_count, net.bridge. bridge-nf-call-iptables, net.ipv4.conf.all. forwarding, net.ipv4.ip_forward, and vm.overcommit_memory.
swapoff	Used to disable paging spaces to improve performance.
rm	Used to clean up residual files in case of deployment failure or other scenarios.
kill	Used to reload Docker processes with updated configuration files.
mkdir chown	Used to install Docker and components. Docker is temporarily installed in the user's /home directory and finally installed in the /usr/bin directory. The components are installed in the project mounting directory you specified, and docker data is stored in the same directory.
groupadd gpasswd	Used to add the Docker user group and add the current user to the Docker user group.
modprobe	Used to add the br_netfilter module required for Docker network communication. You only need to use the modprobe command if the br_netfilter module is not loaded.
systemctl	Used to control the docker service to start.
sudo	Used to enable the use of the sudo -l command to check if a command has sudo permissions.

Edit the sudoers file in the /etc directory. Ensure to perform the follwoing two steps.

Step	Description
Modify the user's sudo permissions.	1. Allow the user account (your username) to execute subsequent configuration commands with sudo permissions as a root user on any host computer. `Username` `ALL=(root) /bin/sh,/bin/mkdir,/bin/rm,/bin/cp, /bin/systemctl,/bin/kill,/usr/sbin/sysctl,/usr/bin/gpasswd, /usr/sbin/groupadd,/usr/bin/chown,/usr/sbin/modprobe, /usr/bin/echo,/ usr/bin/sed,/usr/sbin/swapoff,/bin/sudo` 2. (Optional) Configure sudo permissions on NOPASSWD if you still lack permissions. `Username` `ALL=(root)NOPASSWD: (root) /bin/sh, /bin/mkdir,/bin/rm,/bin/cp,/bin/systemctl,/bin/kill, /usr/sbin/sysctl,/usr/bin/gpasswd,/usr/sbin/groupadd, /usr/bin/chown,/usr/sbin/modprobe,/usr/bin/echo, /usr/bin/sed,/usr/sbin/swapoff,/bin/sudo`
Disable the requiretty option for the user.	Comment out the original requiretty-related content. Ensure that your username does not require a TTY session to execute sudo commands. Otherwise, you cannot use sudo commands through scripts/remote commands. #Defaults: requiretty

Step

Description

Modify the user's sudo permissions.

1. Allow the user account (your username) to execute subsequent configuration commands with sudo permissions as a root user on any host computer.

Username     ALL=(root) /bin/sh,/bin/mkdir,/bin/rm,/bin/cp,
/bin/systemctl,/bin/kill,/usr/sbin/sysctl,/usr/bin/gpasswd,
/usr/sbin/groupadd,/usr/bin/chown,/usr/sbin/modprobe,
/usr/bin/echo,/ usr/bin/sed,/usr/sbin/swapoff,/bin/sudo

2. (Optional) Configure sudo permissions on NOPASSWD if you still lack permissions.

Username     ALL=(root)NOPASSWD: (root) /bin/sh,
/bin/mkdir,/bin/rm,/bin/cp,/bin/systemctl,/bin/kill,
/usr/sbin/sysctl,/usr/bin/gpasswd,/usr/sbin/groupadd,
/usr/bin/chown,/usr/sbin/modprobe,/usr/bin/echo,
/usr/bin/sed,/usr/sbin/swapoff,/bin/sudo

Disable the requiretty option for the user.

Comment out the original requiretty-related content.

Ensure that your username does not require a TTY session to execute sudo commands. Otherwise, you cannot use sudo commands through scripts/remote commands.

#Defaults: requiretty

The following is an example of the sudoers file.

Note:

The content is for reference only. Do not use it to overwrite the original file content. Modify the items one by one referring to the above steps.

# User privilege specification
root    ALL=(ALL:ALL)       ALL

# Defaults requiretty

# Allow the user to run specific commands as root
Username     ALL=(root) /bin/sh,/bin/mkdir,/bin/rm,/bin/cp,/bin/systemctl,/bin/kill,
/usr/sbin/sysctl,/usr/bin/gpasswd,/usr/sbin/groupadd,/usr/bin/chown,
/usr/sbin/modprobe,/usr/bin/echo,/usr/bin/sed,/usr/sbin/swapoff,/bin/sudo

Recommended Configuration

This section provides recommended deployment plans and server configurations based on the common number of daily active users and tables.

Note:

1. For FineOps-deployed FineBI projects, a master component must be deployed. The bi-web and worker components can only be deployed in a ratio of 1:1 or 1:2. The node number ratio cannot be customized.

2. Increase the number of engine nodes if you need to split the read and write operations. To ensure the highly available service, the number of engine nodes must be at least two.

Number of Daily Active Users	Number of Tables (including basic tables and analysis tables)	Deployment Plan	Server Configuration (The component server and the FineOps server can be the same server in case of insufficient resources.)
Less than 100	10,000 tables 10 million rows of data	One bi-web component, one worker component, one master component, and one bi-minio component	Main application server: One, 16 cores, 64 GB, used to deploy one bi-web component, one worker component, one master component, and one bi-minio component Component server (The component server and the FineOps server can be the same server): One, 8 cores, 32 GB, used to deploy the nginx component, the configuration database, the elasticsearch component, and the cluster component
200	10,000 tables 10 million rows of data	Two bi-web components, two worker components, one master component, and one bi-mino component	Main application server: One, 16 cores, 64 GB, used to deploy one bi-web component, one worker component, one master component, and one bi-minio component One, 16 cores, 64 GB, used to deploy one bi-web component and one worker component Component server: One, 8 cores, 32 GB, used to deploy the nginx component, the configuration database, the elasticsearch component, and the cluster component
500	10,000 tables 10 million rows of data	Two bi-web components, two worker components, one master component, and one bi-mino component	Main application server: One, 24 cores, 96 GB, used to deploy one bi-web component, one worker component, one master component, and one bi-minio component One, 24 cores, 96 GB, used to deploy one bi-web component and one worker component Component server: One, 12 cores, 48 GB, used to deploy the nginx component, the configuration database, the elasticsearch component, and the cluster component
1,000	10,000 tables 10 million rows of data	Two bi-web components, two worker components, one master component, and one bi-mino component	Main application server: One, 32 cores, 128 GB, used to deploy one bi-web component, one worker component, one master component, and one bi-minio component One, 32 cores, 128 GB, used to deploy one bi-web component and one worker component Component server: One, 16 cores, 64 GB, used to deploy the nginx component, the configuration database, the elasticsearch component, and the cluster component
2,000	50,000 tables 100 million rows of data	Three bi-web components, three worker components, one master component, and one bi-mino component	Main application server: One, 48 cores, 192 GB, used to deploy one bi-web component, one worker component, one master component, and one bi-minio component One, 48 cores, 192 GB, used to deploy one bi-web component and one worker component One, 48 cores, 192 GB, used to deploy one bi-web component and one worker component Component server: One, 24 cores, 96 GB, used to deploy the nginx component, the configuration database, the elasticsearch component, and the cluster component
4,000	100,000 tables 100 million rows of data	Three bi-web components, three worker components, one master component, and one bi-mino component	Main application server: One, 64 cores, 256 GB, used to deploy one bi-web component, one worker component, one master component, and one bi-minio component One, 64 cores, 256 GB, used to deploy one bi-web component and one worker component One, 64 cores, 256 GB, used to deploy one bi-web component and one worker component Component server: One, 32 cores, 128 GB, used to deploy the nginx component, the configuration database, the elasticsearch component, and the cluster component

Number of Daily Active Users

Number of Tables (including basic tables and analysis tables)

Deployment Plan

Server Configuration (The component server and the FineOps server can be the same server in case of insufficient resources.)

Less than 100

10,000 tables

10 million rows of data

One bi-web component, one worker component, one master component, and one bi-minio component

Main application server:

One, 16 cores, 64 GB, used to deploy one bi-web component, one worker component, one master component, and one bi-minio component

Component server (The component server and the FineOps server can be the same server):

One, 8 cores, 32 GB, used to deploy the nginx component, the configuration database, the elasticsearch component, and the cluster component

200

10,000 tables

10 million rows of data

Two bi-web components, two worker components, one master component, and one bi-mino component

Main application server:

One, 16 cores, 64 GB, used to deploy one bi-web component, one worker component, one master component, and one bi-minio component

One, 16 cores, 64 GB, used to deploy one bi-web component and one worker component

Component server:

One, 8 cores, 32 GB, used to deploy the nginx component, the configuration database, the elasticsearch component, and the cluster component

500

10,000 tables

10 million rows of data

Two bi-web components, two worker components, one master component, and one bi-mino component

Main application server:

One, 24 cores, 96 GB, used to deploy one bi-web component, one worker component, one master component, and one bi-minio component

One, 24 cores, 96 GB, used to deploy one bi-web component and one worker component

Component server:

One, 12 cores, 48 GB, used to deploy the nginx component, the configuration database, the elasticsearch component, and the cluster component

1,000

10,000 tables

10 million rows of data

Two bi-web components, two worker components, one master component, and one bi-mino component

Main application server:

One, 32 cores, 128 GB, used to deploy one bi-web component, one worker component, one master component, and one bi-minio component

One, 32 cores, 128 GB, used to deploy one bi-web component and one worker component

Component server:

One, 16 cores, 64 GB, used to deploy the nginx component, the configuration database, the elasticsearch component, and the cluster component

2,000

50,000 tables

100 million rows of data

Three bi-web components, three worker components, one master component, and one bi-mino component

Main application server:

One, 48 cores, 192 GB, used to deploy one bi-web component, one worker component, one master component, and one bi-minio component

One, 48 cores, 192 GB, used to deploy one bi-web component and one worker component One, 48 cores, 192 GB, used to deploy one bi-web component and one worker component

Component server:

One, 24 cores, 96 GB, used to deploy the nginx component, the configuration database, the elasticsearch component, and the cluster component

4,000

100,000 tables

100 million rows of data

Three bi-web components, three worker components, one master component, and one bi-mino component

Main application server:

One, 64 cores, 256 GB, used to deploy one bi-web component, one worker component, one master component, and one bi-minio component

One, 64 cores, 256 GB, used to deploy one bi-web component and one worker component One, 64 cores, 256 GB, used to deploy one bi-web component and one worker component

Component server:

One, 32 cores, 128 GB, used to deploy the nginx component, the configuration database, the elasticsearch component, and the cluster component

The configuration requirements in the above table must be met because each component of FineBI projects has minimum operational requirements, as shown in the following table.

Component	Maximum CPU Usage	Maximum Memory Usage	Memory Allocation (Xmx)	Disk Space
bi-web	16 cores	18 GB	12 GB	100 GB
bi-engine-worker	16 cores	30 GB	16 GB	100 GB
bi-engine-master	4 cores	8 GB	6 GB	100 GB
bi-minio (data storage)	2 cores	4 GB	/	200 GB
ops_agent (O&M component)	1 core	/	1 GB	/
filebeat (O&M component)	0.1 core	0.1 GB	/	/
elasticsearch (log service)	1 core	/	2 GB	50 GB
nginx (internal gateway)	1 core	/	/	/
mysql (configuration database)	2 cores	/	/	50 GB
redis (state service)	1 core	/	4 GB	/
minio (file service)	1 core	/	/	100 GB

Helpful
Not helpful
Only read

中文（繁體）中文（简体）

English

Server Configuration Confirmation

Overview

Minimum Configuration

Basic Configuration

User Permission

Recommended Configuration

附件列表