The Maximum Number of Open Files on Linux

When deploying FineBI on a Linux server, sometimes an error may occur indicating that there are too many open files.

This is because the maximum number of open files in Linux system is 1024 by default. When BI is executed, it reads data saved locally. In some cases, the number of files opened by BI may exceed this limit. Therefore, the maximum number of open files in Linux system needs to be manually changed. The modification will involve three key values in different situations.

Related Parameter Description

1. nofile

nofile value refers to the maximum number of open files for a single process.

2. nr_open

It refers to the maximum number of files that can be allocated to a single process, usually with a default value of 1024 * 1024 = 1048576.

3. file-max

It is the maximum number of open files that the system kernel can handle, with a default value of 185745.

How to Modify

In general, the value of nofile cannot exceed the values of nr_open and file-max. Therefore, when modifying nofile, you need to consider whether its value exceeds the values of nr_open and file-max.

• When the nofile value (the maximum number of open files) to be modified does not exceed the values of nr_open and file-max, you can directly modify it.

• When the nofile value (the maximum number of open files) to be modified exceeds the values of nr_open and file-max, you need to modify both the nofile value and the values of nr_open and file-max.

Viewing Related Parameter Size

View the size of the parameters before modification.

Command is as follows:

cat /proc/sys/fs/nr_open
cat /proc/sys/fs/file-max
ulimit -n

Modifying the nofile Value in the limit

When the nofile value to be modified (the maximum number of open files) does not exceed the values of nr_open and file-max, you can directly modify it.

Taking Effect Temporarily

Enter ulimit -n 128000 as follows:

Taking Effect Permanently

Note: To allow the changes taking effect immediately, you can see the last step of this section and directly execute reboot to restart the server. If you do not want to restart the server, you can refer to the complete steps in this section.

1. Use vi /etc/security/limits.conf to edit the file limits.conf and modify the nofile value (128000).

vi /etc/security/limits.conf # Go to the file editing interface.
 * soft nofile 128000 # Make sure there is this line.
 * hard nofile 128000 # Make sure there is this line.

Note: You need to replace "*" with a specific username in some systems to allow changes taking effect, such as root soft nofile 128000.

2. Edit the /etc/pam.d/login file to ensure that it contains the following content.

session required pam_limits.so

3. Use the exit command to exit the terminal and log in again. 如下图所示：

nr_open and file-max Values

Taking Effect Temporarily

echo 1200000 > /proc/sys/fs/nr_openecho
200000 > /proc/sys/fs/file-max

Taking Effect Permanently (Restarting the Server)

When the maximum number of open files to be modified exceeds the value of nr_open and file-max, it is necessary to increase this value.

Add the following code to the file /proc/sys/fs/nr_open: (1200000 is the modified parameter value.)

fs.nr_open=1200000

Insert the following code into the file /proc/sys/fs/file-max:

fs.file-max=200000

Save and execute reboot to restart the server.

Note: The kernel parameter nr_open of Linux can only be set in versions after 2.6.25. To check the kernel version, you can use the command uname -a.

Then modify the value of nofile. You can see the section "Modifying the nofile Value in the limit".

Taking Effect Permanently (Not Restarting the Server)

Set fs.nr_open=1200000 fs.file-max=200000 in /etc/sysctl.conf and execute sysctl -p to take effect.

Preview

Use cat /proc/pid/limits to view the maximum number of open files of a single process.

Ineffective Modification for Ordinary Users

1. Problem

After following the steps in the section "Taking Effect Permanently", you may find that your modifications do not take effect. An error occurrs when you use ssh to connect to Linux: ulimit: open files: cannot modify limit: Operation not permitted.

2. Cause

This problem is found after upgrading openssh and only when you log in ssh. So it is related to ssh.

3. Solution

a. Modify the sshd_config file and change #UseLogin no to UseLogin yes.

vi  /etc/ssh/sshd_config

b. Use service sshd restart to restart ssh.