[Dovecot] dovecot: imap-login: Error: net_connect_unix(imap) failed: Resource temporarily unavailable
We recently switched over mail servers to one running dovecot and postfix on RedHat Enterprise 6. The mail store is NFS on a NetApp filer, the index files are on local disk. We have about 6000 total active users, though not all of them access the system at the same time.
All goes well for a while, then we start seeing errors like this in the log:
dovecot: imap-login: Error: net_connect_unix(imap) failed: Resource temporarily unavailable imap-login: Internal login failure (auth failed, 1 attempts)
And at this point, users can't log in. I don't know what Resource dovecot is running out of.
The machine is a 12-core Xeon 2.27Ghz, with 24GB RAM.
I have removed the 1024 nproc limit in /etc/security/limits.d/90-nproc.conf, and upped the nproc and nofile limits to 16384 and 65535 respectively, both in limits.conf, as well as adding ulimit statements in /etc/init.d/dovecot (belt and suspenders).
I have tried adjusting dovecot.conf, but I'm obviously missing something.
dovecot -n follows:
# 2.0.beta6 (3156315704ef): /etc/dovecot/dovecot.conf # OS: Linux 2.6.32-71.14.1.el6.x86_64 x86_64 Red Hat Enterprise Linux Server release 6.0 (Santiago) auth_master_user_separator = * auth_mechanisms = plain login default_client_limit = 5000 default_process_limit = 5000 disable_plaintext_auth = no mail_fsync = always mail_gid = 501 mail_location = maildir:~/Maildir:INDEX=/var/indexes/%n mail_nfs_storage = yes mail_plugins = quota mail_uid = 501 passdb { args = /etc/dovecot/passwd.masterusers driver = passwd-file master = yes pass = yes } passdb { args = /etc/dovecot/dovecot-ldap.conf driver = ldap } plugin { quota = maildir:User quota quota_rule = *:storage=200M } service auth { client_limit = 32768 unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0660 user = postfix } user = root } service imap-login { process_min_avail = 12 service_count = 0 } service pop3-login { service_count = 0 } shutdown_clients = no ssl_cert = </etc/pki/postfix/certs/nexus.pem ssl_key = </etc/pki/postfix/private/nexus_key.pem userdb { args = /etc/dovecot/dovecot-ldap.conf driver = ldap } protocol imap { mail_plugins = $mail_plugins imap_quota }
Thanks, -Kurt Hockenbury
- Kurt Hockenbury <khockenb@stevens.edu>:
We recently switched over mail servers to one running dovecot and postfix on RedHat Enterprise 6. The mail store is NFS on a NetApp filer, the index files are on local disk. We have about 6000 total active users, though not all of them access the system at the same time.
That sounds nice.
# 2.0.beta6 (3156315704ef): /etc/dovecot/dovecot.conf
Is this really a 2.0 beta?
-- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt@charite.de | http://www.charite.de
- Kurt Hockenbury <khockenb@stevens.edu>:
On Tue, 1 Feb 2011, Ralf Hildebrandt wrote:
# 2.0.beta6 (3156315704ef): /etc/dovecot/dovecot.conf
Is this really a 2.0 beta?
That's what is shipping with RHEL 6. We've been trying to keep the system as close to stock RH as possible, to make support easier.
In that case I'd really, really, really use a proper 2.0.x release. 2.0.9 has had soooo many fixes. After all, that's your MAIN application on that box.
-- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt@charite.de | http://www.charite.de
On Tue, 1 Feb 2011, Ralf Hildebrandt wrote:
In that case I'd really, really, really use a proper 2.0.x release. 2.0.9 has had soooo many fixes. After all, that's your MAIN application on that box.
Thanks, I'll upgrade to either 2.0.9 or 2.0.8-3 (the most recent RPM on ATrpms.net) tomorrow morning.
-Kurt
El Tuesday 01 February 2011, Kurt Hockenbury <khockenb@stevens.edu> dijo:
That's what is shipping with RHEL 6. We've been trying to keep the system as close to stock RH as possible, to make support easier.
We are in a similar situation (using RH and not moving too much from that), but using an up to date version of dovecot. After all, when you are having problems with dovecot you're not calling redhat for support. If you're coming to this list, it makes more sense to use the version this list recommends.
HTH
Joseba Torre. Vicegerencia de TICs, área de Explotación
On Tue, 2011-02-01 at 11:31 -0500, Kurt Hockenbury wrote:
dovecot: imap-login: Error: net_connect_unix(imap) failed: Resource temporarily unavailable imap-login: Internal login failure (auth failed, 1 attempts)
I did a lot of fixes related to this and since I haven't heard of it happening for a while now with latest 2.0.x versions, I think I managed to fix it..
# 2.0.beta6 (3156315704ef): /etc/dovecot/dovecot.conf
Unfortunately RHEL6 was frozen early during v2.0 development before all the bugs were fixed. RHEL6.1's Dovecot is hopefully a pretty good one. Until then, I'm not sure what you can do to avoid the above problem, other than upgrade to a newer v2.0. You could try if this helps:
service imap { process_min_avail = 12 }
Ok, we updated to dovecot 2.0.9, and we're still seeing the "net_connect_unix(imap) failed" problem happen around peak login time, 9-10am.
dovecot -n # 2.0.9: /etc/dovecot/dovecot.conf # OS: Linux 2.6.32-71.14.1.el6.x86_64 x86_64 Red Hat Enterprise Linux Server release 6.0 (Santiago) auth_master_user_separator = * auth_mechanisms = plain login default_client_limit = 5000 default_process_limit = 5000 disable_plaintext_auth = no mail_fsync = always mail_gid = 501 mail_location = maildir:~/Maildir:INDEX=/var/indexes/%n mail_nfs_storage = yes mail_plugins = quota mail_uid = 501 passdb { args = /etc/dovecot/passwd.masterusers driver = passwd-file master = yes pass = yes } passdb { args = /etc/dovecot/dovecot-ldap.conf driver = ldap } plugin { quota = maildir:User quota quota_rule = *:storage=200M } service anvil { client_limit = 15000 } service auth { client_limit = 32768 unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0660 user = postfix } user = root } service imap-login { process_min_avail = 12 service_count = 0 } service pop3-login { service_count = 0 } shutdown_clients = no ssl_cert = </etc/pki/postfix/certs/nexus.pem ssl_key = </etc/pki/postfix/private/nexus_key.pem userdb { args = /etc/dovecot/dovecot-ldap.conf driver = ldap } protocol imap { mail_plugins = quota imap_quota }
W dniu 2011-02-03 16:33, Kurt Hockenbury pisze:
Ok, we updated to dovecot 2.0.9, and we're still seeing the "net_connect_unix(imap) failed" problem happen around peak login time, 9-10am.
dovecot -n # 2.0.9: /etc/dovecot/dovecot.conf # OS: Linux 2.6.32-71.14.1.el6.x86_64 x86_64 Red Hat Enterprise Linux Server release 6.0 (Santiago) auth_master_user_separator = * auth_mechanisms = plain login default_client_limit = 5000 default_process_limit = 5000 disable_plaintext_auth = no mail_fsync = always mail_gid = 501 mail_location = maildir:~/Maildir:INDEX=/var/indexes/%n mail_nfs_storage = yes mail_plugins = quota mail_uid = 501 passdb { args = /etc/dovecot/passwd.masterusers driver = passwd-file master = yes pass = yes } passdb { args = /etc/dovecot/dovecot-ldap.conf driver = ldap } plugin { quota = maildir:User quota quota_rule = *:storage=200M } service anvil { client_limit = 15000 } service auth { client_limit = 32768 unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0660 user = postfix } user = root } service imap-login { process_min_avail = 12 service_count = 0 } service pop3-login { service_count = 0 } shutdown_clients = no ssl_cert = </etc/pki/postfix/certs/nexus.pem ssl_key = </etc/pki/postfix/private/nexus_key.pem userdb { args = /etc/dovecot/dovecot-ldap.conf driver = ldap } protocol imap { mail_plugins = quota imap_quota }
This is also my problem. See this: http://www.mail-archive.com/dovecot@dovecot.org/msg36356.html
Len7hir
On Fri, 2011-02-04 at 11:13 +0100, Len7hir wrote:
W dniu 2011-02-03 16:33, Kurt Hockenbury pisze:
Ok, we updated to dovecot 2.0.9, and we're still seeing the "net_connect_unix(imap) failed" problem happen around peak login time, 9-10am. .. This is also my problem. See this: http://www.mail-archive.com/dovecot@dovecot.org/msg36356.html
It might be the same problem, but most likely yours has to do with auth process settings. You haven't given doveconf -n output yet so I can't really guess.
W dniu 2011-02-04 18:26, Timo Sirainen pisze:
On Fri, 2011-02-04 at 11:13 +0100, Len7hir wrote:
W dniu 2011-02-03 16:33, Kurt Hockenbury pisze:
Ok, we updated to dovecot 2.0.9, and we're still seeing the "net_connect_unix(imap) failed" problem happen around peak login time, 9-10am. .. This is also my problem. See this: http://www.mail-archive.com/dovecot@dovecot.org/msg36356.html
It might be the same problem, but most likely yours has to do with auth process settings. You haven't given doveconf -n output yet so I can't really guess.
# 2.0.9: /etc/dovecot/dovecot.conf # OS: Linux 2.6.30.1-m5r2-bond i686 Debian 4.0 auth_mechanisms = plain login default_client_limit = 4096 default_process_limit = 2000 disable_plaintext_auth = no listen = * login_greeting = mail_location = passdb { args = driver = our_driver } protocols = imap service auth { client_limit = 6048 service_count = 0 } service imap-login { process_min_avail = 8 service_count = 0 vsz_limit = 128 M } service imap { process_limit = 1024 process_min_avail = 8 } service pop3-login { inet_listener pop3 { ssl = no } process_min_avail = 8 service_count = 0 } service pop3 { process_limit = 1024 process_min_avail = 8 } ssl_cert = </etc/ssl/certs/pop3.pem ssl_key = </etc/ssl/certs/pop3.pem syslog_facility = local4 userdb { driver = onet } protocol imap { ssl_ca = </etc/ssl/certs/intermediate.geotrust.com ssl_cert = </etc/ssl/certs/imap.cert ssl_key = </etc/ssl/certs/imap.key } protocol pop3 { ssl_cert = </etc/ssl/certs/pop3.pem ssl_key = </etc/ssl/certs/pop3.pem }
-- Len7hir
On 7.2.2011, at 16.35, Len7hir wrote:
passdb { args = driver = our_driver } userdb { driver = onet }
I'm guessing your driver isn't doing lookups fast enough for Dovecot. There is only one auth master process and it needs to do the user lookups really fast (typically asynchronously). You could change them pretty easily to use auth worker processes though. In passdb and userdb preinit() set:
module->module.blocking = TRUE;
On Thu, 2011-02-03 at 10:33 -0500, Kurt Hockenbury wrote:
Ok, we updated to dovecot 2.0.9, and we're still seeing the "net_connect_unix(imap) failed" problem happen around peak login time, 9-10am.
Hmm. Maybe I need to do something more complex here to make it wait for a longer time to be able to connect.. You could try if this helps though:
service imap { process_min_avail = 12 }
On Fri, 4 Feb 2011, Timo Sirainen wrote:
On Thu, 2011-02-03 at 10:33 -0500, Kurt Hockenbury wrote:
Ok, we updated to dovecot 2.0.9, and we're still seeing the "net_connect_unix(imap) failed" problem happen around peak login time, 9-10am.
Hmm. Maybe I need to do something more complex here to make it wait for a longer time to be able to connect.. You could try if this helps though:
service imap { process_min_avail = 12 }
I added that to the config, but we still saw the problem re-occur starting around 9:45am and again at 2pm.
dovecot: imap-login: Error: net_connect_unix(imap) failed: Resource temporarily unavailable
Right now, the only way we seem to be able to recover is to restart dovecot, which isn't a good solution.
-Kurt
On Mon, 2011-02-07 at 17:32 -0500, Kurt Hockenbury wrote:
dovecot: imap-login: Error: net_connect_unix(imap) failed: Resource temporarily unavailable
Right now, the only way we seem to be able to recover is to restart dovecot, which isn't a good solution.
You mean it keeps giving this error without automatically going away? I thought it would only be happening temporarily. Do you see these messages in your logs:
master: Warning: service(imap): process_limit reached, client connections are being dropped
On Tue, 8 Feb 2011, Timo Sirainen wrote:
You mean it keeps giving this error without automatically going away? I thought it would only be happening temporarily. Do you see these messages in your logs:
master: Warning: service(imap): process_limit reached, client connections are being dropped
Aha! Yes, I do. Sorry I missed those before, I was grepping for Error, and missed the warning.
However, I have "default_process_limit = 5000" and I'm pretty sure we don't have that many people connecting. I will try increasing it, however.
-Kurt
On 8.2.2011, at 2.39, Kurt Hockenbury wrote:
master: Warning: service(imap): process_limit reached, client connections are being dropped
Aha! Yes, I do. Sorry I missed those before, I was grepping for Error, and missed the warning.
Yeah, I wonder how I could get this error message to show up in the "Resource temporarily unavailable" message itself..
However, I have "default_process_limit = 5000" and I'm pretty sure we don't have that many people connecting. I will try increasing it, however.
imap/pop3 processes don't use the default limit, you have to set them explicitly. This is because the default_process_limit=100 is good for pretty much all processes, except for imap/pop3 which are needed much more.
On 8.2.2011, at 3.15, Timo Sirainen wrote:
On 8.2.2011, at 2.39, Kurt Hockenbury wrote:
master: Warning: service(imap): process_limit reached, client connections are being dropped
Aha! Yes, I do. Sorry I missed those before, I was grepping for Error, and missed the warning.
Yeah, I wonder how I could get this error message to show up in the "Resource temporarily unavailable" message itself..
In v2.0.10 you'll get:
Feb 08 03:29:11 imap-login: Error: read(imap) failed: Remote closed connection (process_limit reached?)
participants (5)
-
Joseba Torre
-
Kurt Hockenbury
-
Len7hir
-
Ralf Hildebrandt
-
Timo Sirainen