Hello,
we have the following problem:
Nov 3 09:43:33 minerva dovecot: [ID 583609 local0.warning] master: Warning: service(anvil): client_limit reached, client connections are being dropped Nov 3 09:51:33 minerva dovecot: [ID 583609 local0.error] imap-login: Error: net_connect_unix(anvil) failed: Connection refused Nov 3 09:51:33 minerva dovecot: [ID 583609 local0.crit] imap-login: Fatal: Couldn't connect to anvil Nov 3 09:52:42 minerva dovecot: [ID 583609 local0.error] pop3-login: Error: net_connect_unix(anvil) failed: Connection refused Nov 3 09:52:42 minerva dovecot: [ID 583609 local0.crit] pop3-login: Fatal: Couldn't connect to anvil Nov 3 09:52:42 minerva dovecot: [ID 583609 local0.error] imap-login: Error: net_connect_unix(anvil) failed: Connection refused Nov 3 09:52:42 minerva dovecot: [ID 583609 local0.crit] imap-login: Fatal: Couldn't connect to anvil
At that time I counted 2030 imap processes from 995 users and 1106 imap-login processes. What can I do to set an appropriate client_limit for anvil?
The configuration is:
# doveadm config -n
# 2.0.5: /opt/local/etc/dovecot/dovecot.conf
# OS: SunOS 5.10 sun4v
auth_verbose = yes
default_client_limit = 3000
default_process_limit = 300
disable_plaintext_auth = no
listen = xxx.xxx.uni-giessen.de localhost
mail_fsync = always
mail_location = mbox:~/Mail:INBOX=/var/mail/%u
mail_nfs_index = yes
mail_nfs_storage = yes
mail_plugins = quota
mmap_disable = yes
namespace {
inbox = yes
location =
prefix =
separator = /
type = private
}
namespace {
hidden = yes
list = no
location =
prefix = Mail/
separator = /
subscriptions = yes
type = private
}
passdb {
driver = pam
}
postmaster_address = postmaster@hrz.uni-giessen.de
protocols = imap pop3
service auth {
client_limit = 5720
}
service imap {
process_limit = 4096
}
ssl_cert =
minerva{root}2510:/
-- Hochschulrechenzentrum der | Mail: Juergen.Obermann@hrz.uni-giessen.de Justus-Liebig-Universitaet | WWW: http://www.uni-giessen.de/obermann Heinrich-Buff-Ring 44 | Tel: 0641-99-13054 (0641-99-13001) D-35392 Giessen, Germany | Fax: 0641-99-13009
On 3.11.2010, at 9.35, Juergen Obermann wrote:
Nov 3 09:43:33 minerva dovecot: [ID 583609 local0.warning] master: Warning: service(anvil): client_limit reached, client connections are being dropped
At that time I counted 2030 imap processes from 995 users and 1106 imap-login processes. What can I do to set an appropriate client_limit for anvil?
Well, it's easy to answer the question:
service anvil { client_limit = 8000 # or something }
But you would probably benefit from not having that many login processes: http://wiki2.dovecot.org/LoginProcess
There are some other weird things going on here though:
default_client_limit = 3000 default_process_limit = 300
If the default process limit is 300 and you haven't overridden that (I didn't see service imap-login in your settings), how do you have 1106 imap-login processes?
Also you didn't mention how many pop3-login processes there were. With anvil's client_limit being 3000, this error shouldn't have happened unless there were a total of about 3000 imap/pop3-login processes.
So maybe there are some bugs related to this.. Also I should add a check to startup that it warns if anvil's client_limit is too low.
Quoting Timo Sirainen tss@iki.fi:
On 3.11.2010, at 9.35, Juergen Obermann wrote:
Nov 3 09:43:33 minerva dovecot: [ID 583609 local0.warning] master: Warning: service(anvil): client_limit reached, client connections are being dropped
At that time I counted 2030 imap processes from 995 users and 1106 imap-login processes. What can I do to set an appropriate client_limit for anvil?
Well, it's easy to answer the question:
service anvil { client_limit = 8000 # or something }
But you would probably benefit from not having that many login processes: http://wiki2.dovecot.org/LoginProcess
I now switched the login processes from high security to high performance mode, because the problem appeared again.
There are some other weird things going on here though:
default_client_limit = 3000 default_process_limit = 300
If the default process limit is 300 and you haven't overridden that (I didn't see service imap-login in your settings), how do you have 1106 imap-login processes?
Perhaps the imap-login processes should have gone away but still hang araound and serve SSL sessions
Also you didn't mention how many pop3-login processes there were. With anvil's client_limit being 3000, this error shouldn't have happened unless there were a total of about 3000 imap/pop3-login processes.
Normally only a few but if dovecot hangs there are up to 64 pop3-login processes (64 is the number of cpus)
So maybe there are some bugs related to this.. Also I should add a check to startup that it warns if anvil's client_limit is too low.
-- Hochschulrechenzentrum der | Mail: Juergen.Obermann@hrz.uni-giessen.de Justus-Liebig-Universitaet | WWW: http://www.uni-giessen.de/obermann/ Heinrich-Buff-Ring 44 | Tel: 0641-99-13054 (0641-99-13001) D-35392 Giessen, Germany | Fax: 0641-99-13009
This message was sent using IMP, the Internet Messaging Program.
On 2010-11-03 9:12 AM, Jürgen Obermann wrote:
Quoting Timo Sirainen tss@iki.fi:
But you would probably benefit from not having that many login processes: http://wiki2.dovecot.org/LoginProcess
I now switched the login processes from high security to high performance mode, because the problem appeared again.
Just curious - how much less secure is the high performance mode?
--
Best regards,
Charles
On Wed, 2010-11-03 at 09:26 -0400, Charles Marcus wrote:
On 2010-11-03 9:12 AM, Jürgen Obermann wrote:
Quoting Timo Sirainen tss@iki.fi:
But you would probably benefit from not having that many login processes: http://wiki2.dovecot.org/LoginProcess
I now switched the login processes from high security to high performance mode, because the problem appeared again.
Just curious - how much less secure is the high performance mode?
Copy&pasting (slightly just updated) from the wiki:
High-performance mode
It works by using a number of long running login processes, each handling a number of connections. This loses much of the security benefits of the login process design, because in case of a security hole (in Dovecot or SSL library) the attacker is now able to see other users logging in and steal their passwords, read their mails, etc.
On Wed, 2010-11-03 at 14:12 +0100, Jürgen Obermann wrote:
default_client_limit = 3000 default_process_limit = 300
If the default process limit is 300 and you haven't overridden that (I didn't see service imap-login in your settings), how do you have 1106 imap-login processes?
Perhaps the imap-login processes should have gone away but still hang araound and serve SSL sessions
No. Looks like default_process_limit was never used. The default was unlimited number of processes. This is now fixed: http://hg.dovecot.org/dovecot-2.0/rev/b4b6bf889044
Removed unnecessary anvil connection from each auth worker process: http://hg.dovecot.org/dovecot-2.0/rev/3ce7879e3ac0
Added a warning to startup if anvil's client_limit is too low: http://hg.dovecot.org/dovecot-2.0/rev/97b286b6ec45
And if verbose_proctitle=yes anvil now shows in ps output how many connections it's currently serving: http://hg.dovecot.org/dovecot-2.0/rev/4d91cec713cf http://hg.dovecot.org/dovecot-2.0/rev/7b1aae5f7bd5
Hopefully those will get rid of this problem in future.
Quoting Timo Sirainen tss@iki.fi:
On Wed, 2010-11-03 at 14:12 +0100, Jürgen Obermann wrote:
default_client_limit = 3000 default_process_limit = 300
If the default process limit is 300 and you haven't overridden that (I didn't see service imap-login in your settings), how do you have 1106 imap-login processes?
Perhaps the imap-login processes should have gone away but still hang araound and serve SSL sessions
No. Looks like default_process_limit was never used. The default was unlimited number of processes. This is now fixed: http://hg.dovecot.org/dovecot-2.0/rev/b4b6bf889044
Removed unnecessary anvil connection from each auth worker process: http://hg.dovecot.org/dovecot-2.0/rev/3ce7879e3ac0
Added a warning to startup if anvil's client_limit is too low: http://hg.dovecot.org/dovecot-2.0/rev/97b286b6ec45
And if verbose_proctitle=yes anvil now shows in ps output how many connections it's currently serving: http://hg.dovecot.org/dovecot-2.0/rev/4d91cec713cf http://hg.dovecot.org/dovecot-2.0/rev/7b1aae5f7bd5
Hopefully those will get rid of this problem in future.
Yes, they do. Thank you! How can I find out what the appropriate limits for our mail-server are?
Greetings, Jürgen
-- Hochschulrechenzentrum der | Mail: Juergen.Obermann@hrz.uni-giessen.de Justus-Liebig-Universitaet | WWW: http://www.uni-giessen.de/obermann/ Heinrich-Buff-Ring 44 | Tel: 0641-99-13054 (0641-99-13001) D-35392 Giessen, Germany | Fax: 0641-99-13009
This message was sent using IMP, the Internet Messaging Program.
On Thu, 2010-11-04 at 20:07 +0100, Jürgen Obermann wrote:
Hopefully those will get rid of this problem in future.
Yes, they do. Thank you! How can I find out what the appropriate limits for our mail-server are?
It should be at least:
service imap-login { process_limit } + service pop3-login { process_limit } + service managesieve-login { process_limit } + 3
- a few more so that during full load it won't temporarily complain about the the limit being reached.
participants (4)
-
Charles Marcus
-
Juergen Obermann
-
Jürgen Obermann
-
Timo Sirainen