Tons of imap-login processes despite client_limit very high
Hi Dovecot community.
We're seeing a ton of imap-login processes running even when using high performance mode (https://doc.dovecot.org/admin_manual/login_processes/#high-performance-mode). According to the docs:
"process_min_avail should be set to be at least the number of CPU cores in the system, so that all of them will be used. Otherwise new processes are created only once an existing one’s connection count reaches client_limit"
We have process_min_avail=4, client_limit=0 and default_client_limit=200000. So we'd expect seeing only 4 imap-login processes serving a ton of connections each. Yet, we see thousands of imap-login processes (more than half of all the imap processes):
$ ps aux | grep imap-login | wc -l 1278 $ ps aux | grep imap | wc -l 2154
We use verbose_proctitle=yes and a there seem to be 2 types of these processes, about half for a single IP and about half we suspect for multiple IPs:
$ ps aux | grep imap-log _apt 1941081 0.0 0.0 10420 8700 ? S 14:57 0:00 dovecot/imap-login [84.115.232.178 TLS proxy] _apt 1941589 0.0 0.0 10532 8648 ? S 14:57 0:00 dovecot/imap-login [119.202.86.160 TLS proxy] _apt 1941789 0.0 0.0 10188 8620 ? S 14:57 0:00 dovecot/imap-login [0 pre-login + 2 TLS proxies] _apt 1942144 0.0 0.0 10716 8748 ? S 14:57 0:00 dovecot/imap-login [0 pre-login + 3 TLS proxies] _apt 1942428 0.0 0.0 10800 8712 ? S 14:57 0:00 dovecot/imap-login [5.41.100.37 TLS proxy] ... $ ps aux | grep imap-log | grep pre-login | wc -l 624 $ ps aux | grep imap-log | grep -v pre-login | wc -l 654
Is having so many imap-login processes normal with our config? Did we misunderstand the docs or is there something wrong here?
default_client_limit = 1048576 default_process_limit = 200000
service imap-login {
client_limit = 0 # default is 0
process_limit = 0 # default is 0
service_count = 100 process_min_avail = 4 vsz_limit = 2G
inet_listener imap { } inet_listener imaps { haproxy = yes port = 994 } }
After further testing we realized that it was due to service_count = 100. We suspect that when the service count is reached, a new process is spawned, explaining the large number of imap-login processes.
With service_count = 0 we stick with only 4 processes (process_min_avail). However, we're concerned with having these processes run indefinitely in case of memory leaks.
Any insights on that?
On 18/07/2023 12:11 EEST D D <pierre.alletru@gmail.com> wrote:
After further testing we realized that it was due to service_count = 100. We suspect that when the service count is reached, a new process is spawned, explaining the large number of imap-login processes.
With service_count = 0 we stick with only 4 processes (process_min_avail). However, we're concerned with having these processes run indefinitely in case of memory leaks.
Any insights on that?
It's unlikely that they leak memory, but in case that worries, you can do two things:
configure vsz_limit for the service(s), or configure service_count for the service(s). The first one will slay the login processes, disconnecting everyone, if there is a leak, the second one will recycle the process at some point.
Since 2.3.19, you can also use https://doc.dovecot.org/settings/core/#core_setting-process_shutdown_filter to shutdown the process if it's memory consumption goes too high.
Aki
Awesome, thank you for the information Aki. :) We'll set vsz_limit and keep service_count = 0, as service_count >= 1 is not an option for us due to the high memory consumption associated with having many imap-login processes running.
The new process_shutdown_filter option looks interesting, though if it shutdowns the process "after finishing the current connections", as the doc says, that may not resolve the issue in our case due to imap-login processes often having lasting connections (preventing the process to be recycled when service_count > 1 in the first place).
Thanks again, very helpful!
participants (2)
-
Aki Tuomi
-
D D