Hello,
we have the following problem:
Nov 3 09:43:33 minerva dovecot: [ID 583609 local0.warning] master: Warning: service(anvil): client_limit reached, client connections are being dropped Nov 3 09:51:33 minerva dovecot: [ID 583609 local0.error] imap-login: Error: net_connect_unix(anvil) failed: Connection refused Nov 3 09:51:33 minerva dovecot: [ID 583609 local0.crit] imap-login: Fatal: Couldn't connect to anvil Nov 3 09:52:42 minerva dovecot: [ID 583609 local0.error] pop3-login: Error: net_connect_unix(anvil) failed: Connection refused Nov 3 09:52:42 minerva dovecot: [ID 583609 local0.crit] pop3-login: Fatal: Couldn't connect to anvil Nov 3 09:52:42 minerva dovecot: [ID 583609 local0.error] imap-login: Error: net_connect_unix(anvil) failed: Connection refused Nov 3 09:52:42 minerva dovecot: [ID 583609 local0.crit] imap-login: Fatal: Couldn't connect to anvil
At that time I counted 2030 imap processes from 995 users and 1106 imap-login processes. What can I do to set an appropriate client_limit for anvil?
The configuration is:
doveadm config -n
2.0.5: /opt/local/etc/dovecot/dovecot.conf
OS: SunOS 5.10 sun4v
auth_verbose = yes default_client_limit = 3000 default_process_limit = 300 disable_plaintext_auth = no listen = xxx.xxx.uni-giessen.de localhost mail_fsync = always mail_location = mbox:~/Mail:INBOX=/var/mail/%u mail_nfs_index = yes mail_nfs_storage = yes mail_plugins = quota mmap_disable = yes namespace { inbox = yes location = prefix = separator = / type = private } namespace { hidden = yes list = no location = prefix = Mail/ separator = / subscriptions = yes type = private } passdb { driver = pam } postmaster_address = postmaster@hrz.uni-giessen.de protocols = imap pop3 service auth { client_limit = 5720 } service imap { process_limit = 4096 } ssl_cert = </etc/ssl/server.crt ssl_key = </etc/ssl/server.key syslog_facility = local0 userdb { driver = passwd } verbose_proctitle = yes version_ignore = yes protocol imap { imap_client_workarounds = delay-newmail tb-extra-mailbox-sep mail_max_userip_connections = 20 mail_plugins = quota imap_quota } protocol pop3 { pop3_client_workarounds = outlook-no-nuls oe-ns-eoh pop3_uidl_format = %08Xv%08Xu } minerva{root}2509:/
grep anvil /var/log/locallog
minerva{root}2510:/
-- Hochschulrechenzentrum der | Mail: Juergen.Obermann@hrz.uni-giessen.de Justus-Liebig-Universitaet | WWW: http://www.uni-giessen.de/obermann Heinrich-Buff-Ring 44 | Tel: 0641-99-13054 (0641-99-13001) D-35392 Giessen, Germany | Fax: 0641-99-13009
On 3.11.2010, at 9.35, Juergen Obermann wrote:
Nov 3 09:43:33 minerva dovecot: [ID 583609 local0.warning] master: Warning: service(anvil): client_limit reached, client connections are being dropped
At that time I counted 2030 imap processes from 995 users and 1106 imap-login processes. What can I do to set an appropriate client_limit for anvil?
Well, it's easy to answer the question:
service anvil { client_limit = 8000 # or something }
But you would probably benefit from not having that many login processes: http://wiki2.dovecot.org/LoginProcess
There are some other weird things going on here though:
default_client_limit = 3000 default_process_limit = 300
If the default process limit is 300 and you haven't overridden that (I didn't see service imap-login in your settings), how do you have 1106 imap-login processes?
Also you didn't mention how many pop3-login processes there were. With anvil's client_limit being 3000, this error shouldn't have happened unless there were a total of about 3000 imap/pop3-login processes.
So maybe there are some bugs related to this.. Also I should add a check to startup that it warns if anvil's client_limit is too low.
Quoting Timo Sirainen <tss@iki.fi>:
On 3.11.2010, at 9.35, Juergen Obermann wrote:
Nov 3 09:43:33 minerva dovecot: [ID 583609 local0.warning] master: Warning: service(anvil): client_limit reached, client connections are being dropped
At that time I counted 2030 imap processes from 995 users and 1106 imap-login processes. What can I do to set an appropriate client_limit for anvil?
Well, it's easy to answer the question:
service anvil { client_limit = 8000 # or something }
But you would probably benefit from not having that many login processes: http://wiki2.dovecot.org/LoginProcess
I now switched the login processes from high security to high performance mode, because the problem appeared again.
There are some other weird things going on here though:
default_client_limit = 3000 default_process_limit = 300
If the default process limit is 300 and you haven't overridden that (I didn't see service imap-login in your settings), how do you have 1106 imap-login processes?
Perhaps the imap-login processes should have gone away but still hang araound and serve SSL sessions
Also you didn't mention how many pop3-login processes there were. With anvil's client_limit being 3000, this error shouldn't have happened unless there were a total of about 3000 imap/pop3-login processes.
Normally only a few but if dovecot hangs there are up to 64 pop3-login processes (64 is the number of cpus)
So maybe there are some bugs related to this.. Also I should add a check to startup that it warns if anvil's client_limit is too low.
-- Hochschulrechenzentrum der | Mail: Juergen.Obermann@hrz.uni-giessen.de Justus-Liebig-Universitaet | WWW: http://www.uni-giessen.de/obermann/ Heinrich-Buff-Ring 44 | Tel: 0641-99-13054 (0641-99-13001) D-35392 Giessen, Germany | Fax: 0641-99-13009
This message was sent using IMP, the Internet Messaging Program.
On 2010-11-03 9:12 AM, Jürgen Obermann wrote:
Quoting Timo Sirainen <tss@iki.fi>:
But you would probably benefit from not having that many login processes: http://wiki2.dovecot.org/LoginProcess
I now switched the login processes from high security to high performance mode, because the problem appeared again.
Just curious - how much less secure is the high performance mode?
--
Best regards,
Charles
On Wed, 2010-11-03 at 09:26 -0400, Charles Marcus wrote:
On 2010-11-03 9:12 AM, Jürgen Obermann wrote:
Quoting Timo Sirainen <tss@iki.fi>:
But you would probably benefit from not having that many login processes: http://wiki2.dovecot.org/LoginProcess
I now switched the login processes from high security to high performance mode, because the problem appeared again.
Just curious - how much less secure is the high performance mode?
Copy&pasting (slightly just updated) from the wiki:
High-performance mode
It works by using a number of long running login processes, each handling a number of connections. This loses much of the security benefits of the login process design, because in case of a security hole (in Dovecot or SSL library) the attacker is now able to see other users logging in and steal their passwords, read their mails, etc.
On Wed, 2010-11-03 at 14:12 +0100, Jürgen Obermann wrote:
default_client_limit = 3000 default_process_limit = 300
If the default process limit is 300 and you haven't overridden that (I didn't see service imap-login in your settings), how do you have 1106 imap-login processes?
Perhaps the imap-login processes should have gone away but still hang araound and serve SSL sessions
No. Looks like default_process_limit was never used. The default was unlimited number of processes. This is now fixed: http://hg.dovecot.org/dovecot-2.0/rev/b4b6bf889044
Removed unnecessary anvil connection from each auth worker process: http://hg.dovecot.org/dovecot-2.0/rev/3ce7879e3ac0
Added a warning to startup if anvil's client_limit is too low: http://hg.dovecot.org/dovecot-2.0/rev/97b286b6ec45
And if verbose_proctitle=yes anvil now shows in ps output how many connections it's currently serving: http://hg.dovecot.org/dovecot-2.0/rev/4d91cec713cf http://hg.dovecot.org/dovecot-2.0/rev/7b1aae5f7bd5
Hopefully those will get rid of this problem in future.
Quoting Timo Sirainen <tss@iki.fi>:
On Wed, 2010-11-03 at 14:12 +0100, Jürgen Obermann wrote:
default_client_limit = 3000 default_process_limit = 300
If the default process limit is 300 and you haven't overridden that (I didn't see service imap-login in your settings), how do you have 1106 imap-login processes?
Perhaps the imap-login processes should have gone away but still hang araound and serve SSL sessions
No. Looks like default_process_limit was never used. The default was unlimited number of processes. This is now fixed: http://hg.dovecot.org/dovecot-2.0/rev/b4b6bf889044
Removed unnecessary anvil connection from each auth worker process: http://hg.dovecot.org/dovecot-2.0/rev/3ce7879e3ac0
Added a warning to startup if anvil's client_limit is too low: http://hg.dovecot.org/dovecot-2.0/rev/97b286b6ec45
And if verbose_proctitle=yes anvil now shows in ps output how many connections it's currently serving: http://hg.dovecot.org/dovecot-2.0/rev/4d91cec713cf http://hg.dovecot.org/dovecot-2.0/rev/7b1aae5f7bd5
Hopefully those will get rid of this problem in future.
Yes, they do. Thank you! How can I find out what the appropriate limits for our mail-server are?
Greetings, Jürgen
-- Hochschulrechenzentrum der | Mail: Juergen.Obermann@hrz.uni-giessen.de Justus-Liebig-Universitaet | WWW: http://www.uni-giessen.de/obermann/ Heinrich-Buff-Ring 44 | Tel: 0641-99-13054 (0641-99-13001) D-35392 Giessen, Germany | Fax: 0641-99-13009
This message was sent using IMP, the Internet Messaging Program.
On Thu, 2010-11-04 at 20:07 +0100, Jürgen Obermann wrote:
Hopefully those will get rid of this problem in future.
Yes, they do. Thank you! How can I find out what the appropriate limits for our mail-server are?
It should be at least:
service imap-login { process_limit } + service pop3-login { process_limit } + service managesieve-login { process_limit } + 3
- a few more so that during full load it won't temporarily complain about the the limit being reached.
participants (4)
-
Charles Marcus
-
Juergen Obermann
-
Jürgen Obermann
-
Timo Sirainen