Upgrade 2.2.27 to 2.3.9.2: master(imap): net_connect_unix(imap) failed: Resource temporarily unavailable

[Eirik Rye]

Hi,

After upgrading Dovecot from version 2.2.27 to 2.3.9.2, we are 
sporadically seeing lots of these errors in the error log on many of our 
servers:

   imap-login: Error: master(imap): net_connect_unix(imap) failed: 
Resource temporarily unavailable - 
http://wiki2.dovecot.org/SocketUnavailable

The issue is causing significant delays and/or timeouts on login.

 From what I can tell, this happens because the imap-login service is 
unable to connect to the imap service socket, however I am unable to 
identify the root cause of the issue.

We have increased the client_limit for the stats service, as described 
in the upgrade document at (https://wiki2.dovecot.org/Upgrading/2.3). It 
is set to the same value as the process_limit for imap service.

System CPU usage does not appear to be saturated. The CPU usage sits at 
around 50% when these errors appear.

We have doubled the number of file descriptors after upgrading from 
2.2.27. It was previously set to 16392 (which worked fine):

~# cat /proc/`pidof dovecot`/limits | grep "open files"
Max open files            30000                30000                files

We have tried increasing default_process_limit and default_client_limit 
from the default 1000 to 3000, but this has no effect.

Current configuration (with irrelevant parts removed):

~# doveconf -n

default_client_limit = 3000 # these were raised after upgrading in 
attempt to remedy
default_process_limit = 3000
default_vsz_limit = 512 M
[...]
service imap-login {
   client_limit = 1250
   inet_listener imap {
     port = 143
   }
   inet_listener imaps {
     port = 993
     ssl = yes
   }
   process_limit = 16
   process_min_avail = 16
   service_count = 0
   vsz_limit = 512 M
}
service imap {
   client_limit = 1
   process_limit = 20000
}
service stats {
   client_limit = 20000
}
[...]

We appear to be nowhere near the 20000 process_limit set for the imap 
service. We are also not seeing any warnings being logged with regards 
to the process_limit.

~# doveadm who -1 | wc -l
8765

~# doveadm process status | grep "^imap " | wc -l
7583

~# ps aux | grep "dovecot/imap" | wc -l
7449

~# doveadm process status | grep -v "^imap "
name       pid   available_count total_count idle_start 
last_status_update last_kill_sent
stats      19875 12384           582387      0          1578305489 
   0
pop3-login 19868 1000            0           0          1578061504 
   0
pop3-login 19867 1000            0           0          1578061503 
   0
pop3-login 19866 1000            0           0          1578061502 
   0
pop3-login 19865 1000            0           0          1578061503 
   0
pop3-login 19864 1000            0           0          1578061503 
   0
pop3-login 19853 1000            0           0          1578061504 
   0
pop3-login 19852 1000            0           0          1578061504 
   0
pop3-login 19851 1000            0           0          1578061504 
   0
pop3-login 19849 1000            0           0          1578061503 
   0
pop3-login 19847 1000            0           0          1578061503 
   0
pop3-login 19846 1000            0           0          1578061503 
   0
pop3-login 19845 1000            0           0          1578061503 
   0
pop3-login 19844 1000            0           0          1578061503 
   0
pop3-login 19843 1000            0           0          1578061503 
   0
pop3-login 19842 1000            0           0          1578061503 
   0
pop3-login 19839 1000            0           0          1578061502 
   0
log        19841 2973            27          0          1578061502 
   0
imap-login 19873 666             14942       0          1578305488 
   0
imap-login 19872 523             32792       0          1578305489 
   0
imap-login 19871 316             62876       0          1578305489 
   0
imap-login 19870 167             78332       0          1578305489 
   0
imap-login 19869 118             97989       0          1578305489 
   0
imap-login 19863 0               184726      0          1578305489 
   0
imap-login 19862 0               193094      0          1578305489 
   0
imap-login 19861 0               186800      0          1578305487 
   0
imap-login 19860 675             17806       0          1578305487 
   0
imap-login 19859 0               169446      0          1578305489 
   0
imap-login 19858 0               143517      0          1578305489 
   0
imap-login 19857 0               119215      0          1578305488 
   0
imap-login 19856 0               151958      0          1578305489 
   0
imap-login 19855 483             47036       0          1578305488 
   0
imap-login 19854 1               185240      0          1578305489 
   0
imap-login 19840 567             23859       0          1578305485 
   0
config     19874 2967            124482      0          1578305489 
   0
auth       19885 2967            124185      0          1578305489 
   0
anvil      10017 967             7567        0          1578305463 
   0

-- 
Eirik