[Dovecot] Dovecot proxying results in "too many open files"

Gianpaolo Del Matto gianpaolo.delmatto at genotec.ch
Wed Aug 6 17:29:24 EEST 2008 

Hi,

Some words about my Dovecot setup:

- 10 dovecot servers
- each server has some local mailboxes
- each server has proxying enabled to seamlessly
	connect to another host if the mailbox is not local

Running Dovecot 1.1.1 on FreeBSD 6.3-RELEASE-p3.

First of all, local mailbox access is fine.
Proxying works also as expected.

Now for the actual problem:

Shortly after startup, Dovecot starts throwing errors concerning
connection which should be proxied.

Here are some examples from the logs:

Aug  6 15:49:10 dovecot: imap-login: socket() failed: Too many open  
files
Aug  6 15:49:10 dovecot: imap-login: proxy(XYZ): connect(1.2.3.4, 143)  
failed: Too many open files
Aug  6 15:49:10 dovecot: imap-login: accept() failed: Too many open  
files
Aug  6 15:49:10 dovecot: imap-login: socket() failed: Too many open  
files
Aug  6 15:49:10 dovecot: imap-login: proxy(XYZ): connect(1.2.3.4, 143)  
failed: Too many open files
Aug  6 15:49:10 dovecot: imap-login: socket() failed: Too many open  
files
Aug  6 15:49:10 dovecot: imap-login: proxy(XYZ): connect(1.2.3.4, 143)  
failed: Too many open files
Aug  6 15:49:10 dovecot: imap-login: socket() failed: Too many open  
files
Aug  6 15:49:10 dovecot: imap-login: proxy(XYZ): connect(1.2.3.4, 143)  
failed: Too many open files
Aug  6 15:49:11 dovecot: imap-login: socket() failed: Too many open  
files
[..]
Aug  6 16:00:05 dovecot: imap-login: accept() failed: Too many open  
files
Aug  6 16:00:08 dovecot: imap-login: accept() failed: Too many open  
files
Aug  6 16:00:11 dovecot: imap-login: accept() failed: Too many open  
files
Aug  6 16:00:13 dovecot: imap-login: accept() failed: Too many open  
files

As from the context it seems to be a file descriptor issue, as such I  
have done
the following:

- added a 'limits -e -n 32768 -U dovecot' to dovecot startup file to  
ensure it has enough FD's
- additionally, I've checked my kernel file limits, however I always  
set it through loader and sysctl to
	be as large as 64k

# sysctl -a |grep files
kern.maxfiles: 65535
kern.maxfilesperproc: 32768
kern.openfiles: 1632

- also checked the limits of the dovecot user like this:

# su -c dovecot root -c 'ulimit -a'
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) 1048576
file size               (blocks, -f) unlimited
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 32768
pipe size            (512 bytes, -p) 1
stack size              (kbytes, -s) 131072
cpu time               (seconds, -t) unlimited
max user processes              (-u) 5547
virtual memory          (kbytes, -v) unlimited

- done some monitoring of file descriptor usage for the dovecot user,  
though somewhat inacurate, like this:

while [ : ]; do date; fstat -u dovecot |wc -l; sleep 1; done

this gave me these results while tailing the logfile:

Wed Aug  6 15:44:12 CEST 2008
      585
Wed Aug  6 15:44:13 CEST 2008
      579
Wed Aug  6 15:44:14 CEST 2008
      582
Wed Aug  6 15:44:15 CEST 2008
      582
Wed Aug  6 15:44:16 CEST 2008
      586
Wed Aug  6 15:44:17 CEST 2008
      585
Wed Aug  6 15:44:18 CEST 2008
      582

So it didn't seem to be anywhere near the theoretical upper limit I've  
set at 32k.

On the other hand, when restarting Dovecot, it works for a few minutes
for at least as long as the FD usage as reported above stays at around  
500.
I remember from the docs that dovecot-login would require the double
amount of file descriptors to run.

Assuming my report is somewhat inaccurate and lags behind, I could  
imagine
that upon a proxy requests it would effectively peak our at some 500 x  
2 FDs.
Thus effectively reaching the 1024 FD barrier.

I conclude this from the fact that I currently have some 350 - 450  
logins in parallel,
most of which can and shell be proxied.


 From the error message in the logs I got the impression, that only  
proxied
connections are affected.
I was not able to reproduce the issue with non-proxied logins to local  
mailboxes.

Some further steps I've done from the configuration side to no avail:

- disabled imaps/pop3s/ssl
- changed between login_process_per_connection yes/no
- played around with login_max_connections et all (raised/lowered)  
limits
- tried with/without plugins enabled
- tried with SQL-based and passwd-file based userdb/passdb, single and  
mixed


Dovecot config:

# dovecot -n
# 1.1.1: /usr/local/etc/dovecot.conf
base_dir: /var/run/dovecot/
protocols: imap pop3
listen: *, [::]
ssl_disable: yes
ssl_cert_file: /usr/local/etc/postfix/tls/server.crt
ssl_key_file: /usr/local/etc/postfix/tls/server.key
disable_plaintext_auth: no
verbose_ssl: yes
login_dir: /var/run/dovecot/login
login_executable(default): /usr/local/libexec/dovecot/imap-login
login_process_per_connection: no
login_greeting_capability(default): yes
login_greeting_capability(imap): yes
login_greeting_capability(pop3): no
login_processes_count: 16
login_max_processes_count: 64
login_max_connections: 64
max_mail_processes: 256
mail_max_userip_connections: 3
verbose_proctitle: yes
mail_privileged_group: mail
mail_uid: 1000
mail_gid: 1000
mail_location: maildir:~/Maildir
fsync_disable: yes
maildir_copy_preserve_filename: yes
mail_drop_priv_before_exec: yes
mail_executable(default): /mailserver/scripts/dovecot/imap
mail_executable(imap): /mailserver/scripts/dovecot/imap
mail_executable(pop3): /mailserver/scripts/dovecot/pop3
mail_plugins: expire mail_log
mail_plugin_dir(default): /usr/local/lib/dovecot/imap
mail_plugin_dir(imap): /usr/local/lib/dovecot/imap
mail_plugin_dir(pop3): /usr/local/lib/dovecot/pop3
mail_log_max_lines_per_sec: 0
imap_client_workarounds(default): delay-newmail netscape-eoh tb-extra- 
mailbox-sep
imap_client_workarounds(imap): delay-newmail netscape-eoh tb-extra- 
mailbox-sep
imap_client_workarounds(pop3):
pop3_no_flag_updates(default): no
pop3_no_flag_updates(imap): no
pop3_no_flag_updates(pop3): yes
pop3_enable_last(default): no
pop3_enable_last(imap): no
pop3_enable_last(pop3): yes
pop3_uidl_format(default): %08Xu%08Xv
pop3_uidl_format(imap): %08Xu%08Xv
pop3_uidl_format(pop3): UID%u-%v
pop3_client_workarounds(default):
pop3_client_workarounds(imap):
pop3_client_workarounds(pop3): outlook-no-nuls oe-ns-eoh
namespace:
   type: private
   inbox: yes
   list: yes
   subscriptions: yes
namespace:
   type: private
   separator: .
   prefix: INBOX.
   hidden: yes
auth default:
   mechanisms: plain login cram-md5
   cache_size: 131072
   username_translation: #@/@%@
   username_format: %Lu
   verbose: yes
   debug: yes
   debug_passwords: yes
   worker_max_request_count: 16384
   passdb:
     driver: sql
     args: /usr/local/etc/dovecot-sql.conf
   userdb:
     driver: prefetch
   userdb:
     driver: passwd-file
     args: username_format=%n /usr/local/etc/postfix/tables/ 
deliver_passwd
   userdb:
     driver: sql
     args: /usr/local/etc/dovecot-sql.conf
   socket:
     type: listen
     client:
       path: /var/spool/postfix/private/auth
       mode: 432
       user: postfix
       group: postfix
     master:
       path: /var/run/dovecot/auth-master
       mode: 384
       user: vpostfix
       group: vpostfix
plugin:
   expire: Trash 14 Trash/* 14 Spam 14 Spam/* 14 VirusAlerts 14  
VirusAlerts/* 14 Quarantine 14 Quarantine/* 14
   expire_dict: proxy::expire
dict:
   expire: mysql:/usr/local/etc/dovecot-expire.conf


Maybe I am missing something here.
I'd really, really appreciate some help on this
to get it sorted out.

Thank you.

Regards,

Gianpaolo

More information about the dovecot mailing list