Max Number of Open Files in Linux

When deploying FineBI using a Linux server, sometimes there is an error reporting too many open filesor too many open files, as shown in the following image.

This problem is due to the fact that the default maximum number of files that can be opened by the Linux system is 1024, and BI will read the data stored locally when it is executed. This change involves 3 key values in different cases. 

1. Description of relevant parameters

1) nofile 

The nofile value is the maximum number of files that can be opened by a single process.

2) nr_open 

value is the maximum number of files that can be allocated to a single process, usually the default value is 1024*1024 = 1048576.

3) file-max 

This value is the maximum value that can be opened by the system kernel in total, the default value is 185745.

2. Modification strategy

In general, the value of nofile is not allowed to exceed the values of nr_open and file-max. Therefore, when modifying nofile, it is important to consider whether these two values are exceeded.

• When the「 nofile」 value (maximum number of open files) to be modified does not exceed the values of 「nr_open」 and「 file-max」, the nofile value can be modified directly.

• When the 「nofile」 value (maximum number of open files) to be modified exceeds the 「nr_open」 and 「file-max」 values, not only should the 「nofile」 value be modified, but also the 「nr_open」 and 「file-max」 values should be modified to meet the 「greater than nofile」 value.
  
  
  

1.Checking the size of relevant parameters

Check the size of the parameters before modification, as shown below.

Command as shown below.

cat /proc/sys/fs/nr_open
cat /proc/sys/fs/file-max
ulimit -n

2 .Modifying the nofile value in limit

When the 「nofile」 value (maximum number of open files) to be modified does not exceed the 「nr_open」 and 「file-max」 values, the 「nofile」 value can be modified directly.

1 )Temporary effect

Type ulimit -n 128000 as shown below.

2)Permanent effect

Note: After referring to step 1 of this section, reboot the server directly to take effect; if you do not want to reboot the server, you can refer to this section for the complete steps

（1) Edit the limits.conf file with vi /etc/security/limits.conf and change the nofile value (where 128000 is the modified nofile value) as follows

vi /etc/security/limits.conf  #Access to the document editing screen
* soft nofile 128000 #Make sure the line is available
* hard nofile 128000  #Make sure the line is available

ote: Some systems require replacing 「*」 with a specific username to take effect, e.g. root soft nofile 128000

（2）Edit the /etc/pam.d/login file to ensure that the following is present, as shown below.

session required pam_limits.so

（3）Just use the exit statement to log out of the terminal and log back in. As shown in the figure below.

3. nr_open and file-max 

1）Temporary effect

echo 1200000 > /proc/sys/fs/nr_open
echo 200000 > /proc/sys/fs/file-max

2) Permanent effect: server restart required

When the maximum number of open files to be modified exceeds nr_open and file-max, the value needs to be changed to increase this value. This is changed as follows.

Add the following code to the file /proc/sys/fs/nr_open: (1200000 is the value of the modified parameter)

fs.nr_open=1200000

Insert the following code in the file /proc/sys/fs/file-max.

fs.file-max=200000

Save and execute reboot to restart the server.

Note: The Linux kernel parameter nr_open can only be set if the kernel version is 2.6.25 or later. To check the kernel version use the command "uname -a".

Then change the nofile value, see section 3.2.

3） Permanent: no need to restart the server

Set fs.nr_open= 1200000 fs.file-max=200000 in /etc/sysctl.conf, then execute sysctl -p to make the configuration take effect. This is shown below.

4. Effects view

Check the maximum number of open files for a single process by cat /proc/pid/limits, as shown below.

1  Normal user changes not in effect

Probelm description

After following the steps in section 3.2.2 of this document, which do not require a server reboot, the changes made by the normal user do not take effect, and connecting to Linux using ssh gives the following error.ulimit: open files: cannot modify limit: Operation not permitted

Analysis

This problem occurred after the openssh upgrade and is only related to ssh logins.

Solution

1）Modify the sshd_config file by changing #UseLogin no to UseLogin yes as follows.

vi  /etc/ssh/sshd_config

2）Use service sshd restart to restart the ssh service.