Hitting wall at 2048 IMAP connections
We are gradually rolling out Dovecot (IMAP only, no POP3) to our
customer base. We are replicating between a pair of CentOS 7 boxes.
All has been working wonderfully. However, to be sure our rollout
continues to go smoothly, we put together a simple benchmark client
program to fire up X persistent IMAP connections (using hundreds of
mailboxes) that login, list the folders, select the INBOX, periodically
FETCH an email from the INBOX, and otherwise sit in an IDLE loop. As
long as we keep the number of concurrent IMAP connections under 2k,
everything works fine. 2,000 processes generate a minimal load on the
server. However, 2048 is a "hard limit" for number of IMAP processes on
a box the way things are configured right now. The number (2048) is a
bit too magical to be anything but a misconfiguration of either some
kernel limit or a dovecot configuration limit. I have been unable to
figure out where else to look. I have pasted in my dovecot config below
along with the kernel setting for max_user_instances.
Hoping someone can tell me what stupid mistake I have made or what else to check.
Just an aside, we front-end this with a separate set of proxy servers that provide an SSL front-end, hence, no SSL in the dovecot config.
/proc/sys/fs/inotify/max_user_instances = 4096
# 2.2.10: /etc/dovecot/dovecot.conf # OS: Linux 3.10.0-123.13.1.el7.x86_64 x86_64 CentOS Linux release 7.0.1406 (Cor e) auth_master_user_separator = * auth_mechanisms = plain login default_internal_user = sysadm default_login_user = sysadm default_vsz_limit = 512 M disable_plaintext_auth = no doveadm_password = ************** doveadm_port = 12345 imap_max_line_length = 128 k listen = * mail_gid = 2001 mail_location = maildir:~/Maildir mail_plugins = notify replication mail_uid = 2001 managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave mbox_write_locks = fcntl namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = }
passdb { args = scheme=PLAIN /z/imap/usr/master-users driver = passwd-file master = yes pass = yes } passdb { args = scheme=PLAIN username_format=%u /z/imap/usr/users driver = passwd-file } plugin { mail_replica = tcp:imap0s sieve = ~/.dovecot.sieve sieve_dir = ~/sieve sieve_max_actions = 32 sieve_max_redirects = 4 sieve_max_script_size = 1M } protocols = imap pop3 sieve service aggregator { fifo_listener replication-notify-fifo { mode = 0666 user = sysadm } unix_listener replication-notify { mode = 0666 user = sysadm } } service auth { unix_listener auth-userdb { mode = 0777 } } service doveadm { inet_listener { port = 12345 } user = sysadm } service imap { process_limit = 4096 } service managesieve-login { inet_listener sieve { port = 4190 } } service managesieve { process_limit = 4096 } service replicator { process_min_avail = 1 } ssl = no userdb { args = username_format=%u /z/imap/usr/users driver = passwd-file } protocol lda { mail_plugins = sieve notify replication } protocol imap { mail_max_userip_connections = 40 } protocol sieve { managesieve_max_line_length = 65536 }
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Tue, 3 Feb 2015, Ron Cleven wrote:
We are gradually rolling out Dovecot (IMAP only, no POP3) to our customer base. We are replicating between a pair of CentOS 7 boxes. All has been working wonderfully. However, to be sure our rollout continues to go smoothly, we put together a simple benchmark client program to fire up X persistent IMAP connections (using hundreds of mailboxes) that login, list the folders, select the INBOX, periodically FETCH an email from the INBOX, and otherwise sit in an IDLE loop. As long as we keep the number of concurrent IMAP connections under 2k, everything works fine. 2,000 processes generate a minimal load on the server. However, 2048 is a "hard limit" for number of IMAP processes on a box the way things are configured right now. The number (2048) is a bit too magical to be anything but a misconfiguration of either some kernel limit or a dovecot configuration limit. I have been unable to figure out where else to look. I have pasted in my dovecot config below along with the kernel setting for max_user_instances.
Hoping someone can tell me what stupid mistake I have made or what else to check.
Just an aside, we front-end this with a separate set of proxy servers that provide an SSL front-end, hence, no SSL in the dovecot config.
/proc/sys/fs/inotify/max_user_instances = 4096
Do you use virtual users? Then this setting is way too low. There should be entries in kernel log or Dovecot.
Steffen Kaiser -----BEGIN PGP SIGNATURE----- Version: GnuPG v1
iQEVAwUBVNDiOHz1H7kL/d9rAQKbjwf+IhLHWBSM7I/G9A5I/RmlpcvbClYdYDlk BrSPaUnKkNFC1Gxm5vzNNnQc1lBlpfhUT+BEQbeB2kUkM2khH4pOt5BJarRYrq8o BjcMrkqIUJuYa/kzPgXpgonhywXWCww5AAA2j2w0I/sjPcoxFcEHrPluPq3CasWx VnB5Y9OLzVeALbuGlmC+/hc+K4SJWdL314hGxC5RTNdTY4bHusF/cUJzv63At3fZ V7ahL77+QQ8elMCmRHFXiT2pCHzx6qw54ZUA6hIDt5VPgsMhrBaF8pwr2LIIHhsz 6VwvMvj5A+N8XH3z2QUVa36V5YvQrYoHVqaO/n+5E/LymcvBXx92QA== =1IfD -----END PGP SIGNATURE-----
On 03.02.2015 15:35, Ron Cleven wrote:
/proc/sys/fs/inotify/max_user_instances = 4096
Maybe you are hitting some file descriptor limit (kernel or ulimit)?
This could help: https://unix.stackexchange.com/questions/36841/why-is-number-of-open-files-l...
On 02/03/2015 03:35 PM, Ron Cleven wrote:
We are gradually rolling out Dovecot (IMAP only, no POP3) to our customer base. We are replicating between a pair of CentOS 7 boxes. All has been working wonderfully. However, to be sure our rollout continues to go smoothly, we put together a simple benchmark client program to fire up X persistent IMAP connections (using hundreds of mailboxes) that login, list the folders, select the INBOX, periodically FETCH an email from the INBOX, and otherwise sit in an IDLE loop. As long as we keep the number of concurrent IMAP connections under 2k, everything works fine. 2,000 processes generate a minimal load on the server. However, 2048 is a "hard limit" for number of IMAP processes on a box the way things are configured right now. The number (2048) is a bit too magical to be anything but a misconfiguration of either some kernel limit or a dovecot configuration limit. I have been unable to figure out where else to look. I have pasted in my dovecot config below along with the kernel setting for max_user_instances.
Hoping someone can tell me what stupid mistake I have made or what else to check.
Just an aside, we front-end this with a separate set of proxy servers that provide an SSL front-end, hence, no SSL in the dovecot config.
You didn't mention but did you try checking process_limit and client_limit for various Dovecot services in the output of doveconf | less ? If I remember correctly, as long as imap process lives, it is a client of anvil process. When anvil reaches it's maximum, it's no longer possible to run more imap processes and therefore no longer possible to connect more clients.
I think I hit some similar problem in the past, but in my case Dovecot complained during startup and warned that anvil's client_limit is too low compared to imap process_limit and that it could lead to problems. So I just increased the limit and was done with it.
That said, this was some time ago so process and setting names can be completely wrong. Hope someone more knowledgeable corrects me in that case
The default connection limit values of dovecot are pathetic - unless you run a home mail server where there is only you, the cat and the dog using it.
OP, lots of editing with trial and error required. from memory massive increases needed to:
service auth -> client limit service imap-login -> process limit service imap -> process limit if using pop3, repeat above two steps s/imap/pop3/ service managesieve -> process limit service anvil -> client_limit (and system ulimit)
I'm on holidays in australia at moment so don't have access to copy and show you a working example for 4K users, i will be back on hong kong in 17 days until then i have very limited access.
On 2/4/15, Jiri Bourek bourek@thinline.cz wrote:
On 02/03/2015 03:35 PM, Ron Cleven wrote:
We are gradually rolling out Dovecot (IMAP only, no POP3) to our customer base. We are replicating between a pair of CentOS 7 boxes. All has been working wonderfully. However, to be sure our rollout continues to go smoothly, we put together a simple benchmark client program to fire up X persistent IMAP connections (using hundreds of mailboxes) that login, list the folders, select the INBOX, periodically FETCH an email from the INBOX, and otherwise sit in an IDLE loop. As long as we keep the number of concurrent IMAP connections under 2k, everything works fine. 2,000 processes generate a minimal load on the server. However, 2048 is a "hard limit" for number of IMAP processes on a box the way things are configured right now. The number (2048) is a bit too magical to be anything but a misconfiguration of either some kernel limit or a dovecot configuration limit. I have been unable to figure out where else to look. I have pasted in my dovecot config below along with the kernel setting for max_user_instances.
Hoping someone can tell me what stupid mistake I have made or what else to check.
Just an aside, we front-end this with a separate set of proxy servers that provide an SSL front-end, hence, no SSL in the dovecot config.
You didn't mention but did you try checking process_limit and client_limit for various Dovecot services in the output of doveconf | less ? If I remember correctly, as long as imap process lives, it is a client of anvil process. When anvil reaches it's maximum, it's no longer possible to run more imap processes and therefore no longer possible to connect more clients.
I think I hit some similar problem in the past, but in my case Dovecot complained during startup and warned that anvil's client_limit is too low compared to imap process_limit and that it could lead to problems. So I just increased the limit and was done with it.
That said, this was some time ago so process and setting names can be completely wrong. Hope someone more knowledgeable corrects me in that case
participants (5)
-
Florian Pritz
-
Jiri Bourek
-
Nick Edwards
-
Ron Cleven
-
Steffen Kaiser