Upgrade 2.2.27 to 2.3.9.2: master(imap): net_connect_unix(imap) failed: Resource temporarily unavailable
Eirik Rye
rye at domeneshop.no
Tue Jan 7 16:15:09 EET 2020
While I not certain that this is related to the problem we are
experiencing, we are also (more rarely) seeing these:
dovecot: master: Warning: service(imap-login): process_limit (16)
reached, client connections are being dropped
imap-login was set to process_limit=16, client_limit=1250 (for a total
of 20000 concurrent clients) on 2.2.27. We never had any issues with
this then.
I tried raising it to client_limit=2500, for a total of 40000 concurrent
connections, but we are still seeing the errors logged every now and then.
Almost every single available imap-login socket is reported as being
used! Is something eating up the sockets processes and not freeing them?
~# doveadm process status | grep "^imap-login "
imap-login 8025 2197 419 0 1578406002
0
imap-login 25790 120 5439 0 1578406002
0
imap-login 24231 8 8573 0 1578406001
0
imap-login 15707 0 91617 0 1578405978
0
imap-login 15706 0 81715 0 1578406002
0
imap-login 15705 0 57033 0 1578405992
0
imap-login 15704 0 88287 0 1578405996
0
imap-login 15703 0 68346 0 1578405999
0
imap-login 15701 0 24825 0 1578406002
0
imap-login 15695 0 15796 0 1578406001
0
imap-login 15693 0 38740 0 1578405997
0
imap-login 15692 0 11609 0 1578405998
0
imap-login 15691 0 46832 0 1578406002
0
imap-login 15690 0 10816 0 1578406001
0
imap-login 15689 0 84235 0 1578406002
0
imap-login 15683 0 30898 0 1578406001
0
I am counting 37 675 used imap-login sockets (according to dovecot), on
a machine with only ~7500 active TCP connections and a similar number of
logged in users:
~# ss -t | wc -l
7430
~# doveadm process status | grep "^imap " | wc -l
7224
Here is `doveconf -n` for service imap-login again (setup in
"high-performance mode" as described in wiki):
service imap-login {
client_limit = 2500
inet_listener imap {
port = 143
}
inet_listener imaps {
port = 993
ssl = yes
}
process_limit = 16
process_min_avail = 16
service_count = 0
vsz_limit = 512 M
}
I would greatly appreciate some pointers, because this is mindboggling
confusing.
On 06/01/2020 11:28, Eirik Rye wrote:
> Hi,
>
> After upgrading Dovecot from version 2.2.27 to 2.3.9.2, we are
> sporadically seeing lots of these errors in the error log on many of our
> servers:
>
> imap-login: Error: master(imap): net_connect_unix(imap) failed:
> Resource temporarily unavailable -
> http://wiki2.dovecot.org/SocketUnavailable
>
> The issue is causing significant delays and/or timeouts on login.
>
> From what I can tell, this happens because the imap-login service is
> unable to connect to the imap service socket, however I am unable to
> identify the root cause of the issue.
>
> We have increased the client_limit for the stats service, as described
> in the upgrade document at (https://wiki2.dovecot.org/Upgrading/2.3). It
> is set to the same value as the process_limit for imap service.
>
> System CPU usage does not appear to be saturated. The CPU usage sits at
> around 50% when these errors appear.
>
> We have doubled the number of file descriptors after upgrading from
> 2.2.27. It was previously set to 16392 (which worked fine):
>
> ~# cat /proc/`pidof dovecot`/limits | grep "open files"
> Max open files 30000 30000 files
>
> We have tried increasing default_process_limit and default_client_limit
> from the default 1000 to 3000, but this has no effect.
>
> Current configuration (with irrelevant parts removed):
>
> ~# doveconf -n
>
> default_client_limit = 3000 # these were raised after upgrading in
> attempt to remedy
> default_process_limit = 3000
> default_vsz_limit = 512 M
> [...]
> service imap-login {
> client_limit = 1250
> inet_listener imap {
> port = 143
> }
> inet_listener imaps {
> port = 993
> ssl = yes
> }
> process_limit = 16
> process_min_avail = 16
> service_count = 0
> vsz_limit = 512 M
> }
> service imap {
> client_limit = 1
> process_limit = 20000
> }
> service stats {
> client_limit = 20000
> }
> [...]
>
> We appear to be nowhere near the 20000 process_limit set for the imap
> service. We are also not seeing any warnings being logged with regards
> to the process_limit.
>
> ~# doveadm who -1 | wc -l
> 8765
>
> ~# doveadm process status | grep "^imap " | wc -l
> 7583
>
> ~# ps aux | grep "dovecot/imap" | wc -l
> 7449
>
> ~# doveadm process status | grep -v "^imap "
> name pid available_count total_count idle_start
> last_status_update last_kill_sent
> stats 19875 12384 582387 0 1578305489 0
> pop3-login 19868 1000 0 0 1578061504 0
> pop3-login 19867 1000 0 0 1578061503 0
> pop3-login 19866 1000 0 0 1578061502 0
> pop3-login 19865 1000 0 0 1578061503 0
> pop3-login 19864 1000 0 0 1578061503 0
> pop3-login 19853 1000 0 0 1578061504 0
> pop3-login 19852 1000 0 0 1578061504 0
> pop3-login 19851 1000 0 0 1578061504 0
> pop3-login 19849 1000 0 0 1578061503 0
> pop3-login 19847 1000 0 0 1578061503 0
> pop3-login 19846 1000 0 0 1578061503 0
> pop3-login 19845 1000 0 0 1578061503 0
> pop3-login 19844 1000 0 0 1578061503 0
> pop3-login 19843 1000 0 0 1578061503 0
> pop3-login 19842 1000 0 0 1578061503 0
> pop3-login 19839 1000 0 0 1578061502 0
> log 19841 2973 27 0 1578061502 0
> imap-login 19873 666 14942 0 1578305488 0
> imap-login 19872 523 32792 0 1578305489 0
> imap-login 19871 316 62876 0 1578305489 0
> imap-login 19870 167 78332 0 1578305489 0
> imap-login 19869 118 97989 0 1578305489 0
> imap-login 19863 0 184726 0 1578305489 0
> imap-login 19862 0 193094 0 1578305489 0
> imap-login 19861 0 186800 0 1578305487 0
> imap-login 19860 675 17806 0 1578305487 0
> imap-login 19859 0 169446 0 1578305489 0
> imap-login 19858 0 143517 0 1578305489 0
> imap-login 19857 0 119215 0 1578305488 0
> imap-login 19856 0 151958 0 1578305489 0
> imap-login 19855 483 47036 0 1578305488 0
> imap-login 19854 1 185240 0 1578305489 0
> imap-login 19840 567 23859 0 1578305485 0
> config 19874 2967 124482 0 1578305489 0
> auth 19885 2967 124185 0 1578305489 0
> anvil 10017 967 7567 0 1578305463 0
>
--
Eirik
More information about the dovecot
mailing list