On 14.2.2012, at 17.04, Christoph Bußenius wrote:
our Dovecot proxy started logging about reaching the client_limit once there were about 1000 connections running over it:
Feb 14 15:22:02 vmmailproxy1 dovecot: master: Warning: service(anvil): client_limit (100000) reached, client connections are being dropped Feb 14 15:22:18 vmmailproxy1 dovecot: master: Warning: service(ipc): client_limit (100000) reached, client connections are being dropped
This indicate some kind of a connection leak.. There shouldn't be so many connections. Each login process is supposed to create only one connection to anvil and one connection to ipc. How many open fds do you see in those processes currently?
I was a bit perplexed because 100.000 is far above 1000. However we found out that in fact the problem was the number of open files, and we have fixed it by putting 'ulimit -n 6000' into the init script.
The number of open files is so large, because there are at least 200k fds used for connections to anvil and ipc processes. Do you see some error message also about "Too many open files", or why do you think the number of open files was the problem? If a process runs out of fds there should be such an error message.
Even though the log claims that connections would be dropped, the number of imap-login processes kept rising above 1000.
Connections to anvil and ipc processes were being dropped. That doesn't necessarily mean that everything else stops.
The "doveadm proxy list" would fail with an error message:
$ sudo doveadm proxy list doveadm(root): Error: LIST failed: Disconnected Segmentation fault
The segfault did not happen every time I tried it. I'm sorry I don't have any core dumps or backtraces, because I was more concerned about putting the proxy into a sane state. Maybe I can reproduce this on a test server if you would like a back trace.
http://hg.dovecot.org/dovecot-2.0/rev/c402fb407501 fixes the crash I think.