[Dovecot] dovecot died
Hello,
today our dovecot imap server died. The following message appeared in the logfile:
dovecot: Mar 23 07:00:58 Error: pipe() failed: Too many open files
We have one dovecot instance for pop3 and one for imap, so while the pop3-service was still available, the imap-service was not running any more.
I've seen a lots of users keeping there imap-connections idle for hours, often more than one per mailbox. How can I limit this? Can I limit the maximum time a connection can be open?
Regards Marten
you may want to try increasing the number of files available for the system. (Assuming your on linux). You'll have to google for the exact location but they should be something like file-max or inode-max under /proc
On Suse... # find /proc/sys | grep file /proc/sys/fs/file-max /proc/sys/fs/file-nr
# cat /proc/sys/fs/file-max 50716
Marten Lehmann wrote:
Hello,
today our dovecot imap server died. The following message appeared in the logfile:
dovecot: Mar 23 07:00:58 Error: pipe() failed: Too many open files
We have one dovecot instance for pop3 and one for imap, so while the pop3-service was still available, the imap-service was not running any more.
I've seen a lots of users keeping there imap-connections idle for hours, often more than one per mailbox. How can I limit this? Can I limit the maximum time a connection can be open?
Regards Marten
Hello,
On Suse... # find /proc/sys | grep file /proc/sys/fs/file-max /proc/sys/fs/file-nr
# cat /proc/sys/fs/file-max 50716
# cat /proc/sys/fs/file-max 102592 # cat /proc/sys/fs/file-nr 19755 0 102592
well, more than 100.000 concurrent filehandles seem to be a lot. But if I'm doing an "lsof", I can find 18 filehandles per imap-process, like:
imap 32703 nobody cwd DIR 104,4 4096 934141 /mailboxes/s imap 32703 nobody rtd DIR 104,2 4096 2 / imap 32703 nobody txt REG 104,4 568944 2572335 /imap/dovecot-1.0.beta3/libexec/dovecot/imap imap 32703 nobody mem REG 104,2 106397 33036 /lib/ld-2.3.4.so imap 32703 nobody mem REG 104,2 1454462 33048 /lib/tls/libc-2.3.4.so imap 32703 nobody mem REG 104,2 15324 33071 /lib/libdl-2.3.4.so imap 32703 nobody 0u unix 0xc4d82880 46130422 socket imap 32703 nobody 1u unix 0xc4d82880 46130422 socket imap 32703 nobody 2w FIFO 0,7 46130423 pipe imap 32703 nobody 3u CHR 5,0 1518 /dev/tty imap 32703 nobody 4r CHR 1,9 440 /dev/urandom imap 32703 nobody 5u CHR 136,6 8 /dev/pts/6 (deleted) imap 32703 nobody 6r FIFO 0,7 46130424 pipe imap 32703 nobody 7w FIFO 0,7 34371569 pipe imap 32703 nobody 8w FIFO 0,7 46130424 pipe imap 32703 nobody 9r FIFO 0,7 46130425 pipe imap 32703 nobody 10w FIFO 0,7 46130425 pipe imap 32703 nobody 11u REG 104,4 296 934811 /mailboxes/s/Maildir/dovecot.index.log
So if some people 100 people are keeping 3 concurrent imap-connections idle (which is default with Mozilla Mail), the 54000 filehandles are opened. And the server also does some nfs-stuff und pop3 and I guess there are more than 100 people keeping connections idle.
So how can I limit this? I can't always wait for the imap-server to die so I can start it again.
Regards Marten Lehmann
Hello,
today dovecot almost died again. Actually, the daemon was still running, but the daemon didn't reply with its greeting message after establishing a connection. I noticed two things that scare me: One dovecot-process was running with almost 100% load and there were two extreme users, one with more than 80 concurrent connections, the other one with about 50 connections.
The only way I found to limit connections was max_mail_processes. But this doesn't help very much. I'm looking for something like MAXPERIP as I find in courier-imap. I don't want to switch back to courier-imap because I cannot limit anything in dovecot.
Regards Marten
you may be able to band-aid things by running dovecot from xinetd and using the 'instances' [0] keyword. I don't know if you can specify 'instances per IP' but it may be in the xinetd website [1]. (ok a litle more digging here [3] - per_source arg) Your connections may get a bit soggy running through xinetd, but I think this will give you some control as a tradeoff.
I wonder how your users are generating that many connections. Try running this shell code [2] at a command prompt and see what the connections are doing (right most column is the timer). Of course, if you are not using ssl, then you problably want to grep for 143, the port number.
[0] - man xinetd.conf
instances determines the number of servers that can be simultaneously active for a service (the default is no limit). The value of this attribute can be either a number or UNLIMITED which means that there is no limit.
[1] - http://www.xinetd.org/
[2] - while [ 1 ]; do netstat -topnavel| grep 993; sleep 2; done
[3] - man xinetd.conf
per_source Takes an integer or "UNLIMITED" as an argument. This specifies the maximum instances of this service per source IP address. This can also be specified in the defaults section.
Good luck Thomas
Marten Lehmann wrote:
Hello,
today dovecot almost died again. Actually, the daemon was still running, but the daemon didn't reply with its greeting message after establishing a connection. I noticed two things that scare me: One dovecot-process was running with almost 100% load and there were two extreme users, one with more than 80 concurrent connections, the other one with about 50 connections.
The only way I found to limit connections was max_mail_processes. But this doesn't help very much. I'm looking for something like MAXPERIP as I find in courier-imap. I don't want to switch back to courier-imap because I cannot limit anything in dovecot.
Regards Marten
Hello,
you may be able to band-aid things by running dovecot from xinetd and using the 'instances' [0] keyword.
no, never again! I'm stigmatised by xinetd. I once tried to run the qmail-pop3-service (which is very simple and not very ressourcen-intensive compared to dovecot-imap) with xinetd and even with setting all options to unlimited, xinetd wasn't able to handle all the connections. Although it should work at any CPU load and no matter how many users connecting, it simply didn't accept any more connections while just 80-90 users were connecting concurrently. So xinetd definetely isn't a choice. (Btw.: I switched to tcpserver from djb to run qmail-pop3d which worked perfectly. But tpcserver doesn't have a connections_per_user option).
I hope Timo will add this feature in the next time, it shouldn't be that hard to track the ip-addresses that connections come from and block them if limits are reached.
Regards Marten
Marten Lehmann
Hello,
today our dovecot imap server died. The following message appeared in
the logfile:dovecot: Mar 23 07:00:58 Error: pipe() failed: Too many open files
We have one dovecot instance for pop3 and one for imap, so while the
pop3-service was still available, the imap-service was not running any more.I've seen a lots of users keeping there imap-connections idle for hours,
often more than one per mailbox. How can I limit this? Can I limit the
maximum time a connection can be open?Regards Marten
You could look at inreasing the file limits, on debian its in /etc/security/limits.conf, increase the default 'nofile' from 1024 to say, 20000 or something.
No idea what it is in other distros, running 'ulimit -n 20000' before starting dovecot (as root) may help too.
Cheers, Trent
Hello,
You could look at inreasing the file limits, on debian its in /etc/security/limits.conf, increase the default 'nofile' from 1024 to say, 20000 or something.
no, thats the wrong way. This would help for now but it is just a workaround and sooner or later even these limits might be exceeded. So dovecot definitely needs max_connections_per_user or max_connections_per_ip option.
Regards Marten
Marten Lehmann wrote:
Hello,
You could look at inreasing the file limits, on debian its in /etc/security/limits.conf, increase the default 'nofile' from 1024 to say, 20000 or something.
no, thats the wrong way. This would help for now but it is just a workaround and sooner or later even these limits might be exceeded. So dovecot definitely needs max_connections_per_user or max_connections_per_ip option.
Here's an example of what we're seeing, haven't pinned it down to a specific client setting yet:
dovecot: Mar 27 13:14:26 Info: imap-login: Login: user=<username1>, method=PLAIN, rip=xxx.xxx.xxx.xxx, lip=10.1.0.102 dovecot: Mar 27 13:14:28 Info: imap-login: Login: user=<username1>, method=PLAIN, rip=xxx.xxx.xxx.xxx, lip=10.1.0.102 dovecot: Mar 27 13:14:30 Info: imap-login: Login: user=<username1>, method=PLAIN, rip=xxx.xxx.xxx.xxx, lip=10.1.0.102 dovecot: Mar 27 13:14:32 Info: imap-login: Login: user=<username1>, method=PLAIN, rip=xxx.xxx.xxx.xxx, lip=10.1.0.102 dovecot: Mar 27 13:14:35 Info: imap-login: Login: user=<username1>, method=PLAIN, rip=xxx.xxx.xxx.xxx, lip=10.1.0.102 dovecot: Mar 27 13:14:36 Info: imap-login: Login: user=<username1>, method=PLAIN, rip=xxx.xxx.xxx.xxx, lip=10.1.0.102 dovecot: Mar 27 13:14:38 Info: imap-login: Login: user=<username1>, method=PLAIN, rip=xxx.xxx.xxx.xxx, lip=10.1.0.102 dovecot: Mar 27 13:14:39 Info: imap-login: Login: user=<username1>, method=PLAIN, rip=xxx.xxx.xxx.xxx, lip=10.1.0.102 dovecot: Mar 27 13:14:41 Info: imap-login: Login: user=<username1>, method=PLAIN, rip=xxx.xxx.xxx.xxx, lip=10.1.0.102 dovecot: Mar 27 13:14:42 Info: imap-login: Login: user=<username1>, method=PLAIN, rip=xxx.xxx.xxx.xxx, lip=10.1.0.102 dovecot: Mar 27 13:14:45 Info: imap-login: Login: user=<username1>, method=PLAIN, rip=xxx.xxx.xxx.xxx, lip=10.1.0.102 dovecot: Mar 27 13:14:46 Info: imap-login: Login: user=<username1>, method=PLAIN, rip=xxx.xxx.xxx.xxx, lip=10.1.0.102 dovecot: Mar 27 13:14:48 Info: imap-login: Login: user=<username1>, method=PLAIN, rip=xxx.xxx.xxx.xxx, lip=10.1.0.102 dovecot: Mar 27 13:14:49 Info: imap-login: Login: user=<username1>, method=PLAIN, rip=xxx.xxx.xxx.xxx, lip=10.1.0.102 dovecot: Mar 27 13:14:51 Info: imap-login: Login: user=<username1>, method=PLAIN, rip=xxx.xxx.xxx.xxx, lip=10.1.0.102 dovecot: Mar 27 13:14:53 Info: imap-login: Login: user=<username1>, method=PLAIN, rip=xxx.xxx.xxx.xxx, lip=10.1.0.102 dovecot: Mar 27 13:14:55 Info: imap-login: Login: user=<username1>, method=PLAIN, rip=xxx.xxx.xxx.xxx, lip=10.1.0.102 dovecot: Mar 27 13:14:56 Info: imap-login: Login: user=<username1>, method=PLAIN, rip=xxx.xxx.xxx.xxx, lip=10.1.0.102 dovecot: Mar 27 13:14:58 Info: imap-login: Login: user=<username1>, method=PLAIN, rip=xxx.xxx.xxx.xxx, lip=10.1.0.102 dovecot: Mar 27 13:14:59 Info: imap-login: Login: user=<username1>, method=PLAIN, rip=xxx.xxx.xxx.xxx, lip=10.1.0.102 dovecot: Mar 27 13:15:01 Info: imap-login: Login: user=<username1>, method=PLAIN, rip=xxx.xxx.xxx.xxx, lip=10.1.0.102 dovecot: Mar 27 13:15:03 Info: imap-login: Login: user=<username1>, method=PLAIN, rip=xxx.xxx.xxx.xxx, lip=10.1.0.102 dovecot: Mar 27 13:15:05 Info: imap-login: Login: user=<username1>, method=PLAIN, rip=xxx.xxx.xxx.xxx, lip=10.1.0.102 dovecot: Mar 27 13:15:06 Info: imap-login: Login: user=<username1>, method=PLAIN, rip=xxx.xxx.xxx.xxx, lip=10.1.0.102 dovecot: Mar 27 13:15:08 Info: imap-login: Login: user=<username1>, method=PLAIN, rip=xxx.xxx.xxx.xxx, lip=10.1.0.102 dovecot: Mar 27 13:15:10 Info: imap-login: Login: user=<username1>, method=PLAIN, rip=xxx.xxx.xxx.xxx, lip=10.1.0.102 dovecot: Mar 27 13:15:12 Info: imap-login: Login: user=<username1>, method=PLAIN, rip=xxx.xxx.xxx.xxx, lip=10.1.0.102 dovecot: Mar 27 13:15:14 Info: imap-login: Login: user=<username1>, method=PLAIN, rip=xxx.xxx.xxx.xxx, lip=10.1.0.102 dovecot: Mar 27 13:15:17 Info: imap-login: Login: user=<username1>, method=PLAIN, rip=xxx.xxx.xxx.xxx, lip=10.1.0.102 dovecot: Mar 27 13:15:18 Info: imap-login: Login: user=<username1>, method=PLAIN, rip=xxx.xxx.xxx.xxx, lip=10.1.0.102 dovecot: Mar 27 13:15:20 Info: imap-login: Login: user=<username1>, method=PLAIN, rip=xxx.xxx.xxx.xxx, lip=10.1.0.102
When we get 1-2 of these going at a time, it kills the server, and we need to restart imap. If there was* some kind of per-ip/login limit, I bet we wouldn't exceed the resources. In this case, we know this is an employee using an unsupported client from outside - but, we have no way to block the IP (need to keep in contact with the remote IP), and there doesn't seem to be a "only accept from these imap clients" option, either... It's totally unacceptable for me to tell my boss the mailserver died because someone used a bad mail program. :/
Maybe this would be really hard to implement in dovecot, but I just wanted to second the notion that it would be a good thing(tm).
Thanks,
-deano
Regards Marten
When we get 1-2 of these going at a time, it kills the server, and we need to restart imap. If there was* some kind of per-ip/login limit, I bet we wouldn't exceed the resources. In this case, we know this is an employee using an unsupported client from outside - but, we have no way to block the IP (need to keep in contact with the remote IP), and there doesn't seem to be a "only accept from these imap clients" option, either... It's totally unacceptable for me to tell my boss the mailserver died because someone used a bad mail program. :/
Maybe this would be really hard to implement in dovecot, but I just wanted to second the notion that it would be a good thing(tm). Whilst this is true, in the mean time, you could use netfilter to limit
Dean Blackburn wrote: <snip> the number of open TCP connections using connlimit from patch-o-matic (base).
http://www.netfilter.org/projects/patch-o-matic/pom-base.html#pom-base-connl...
iptables -p tcp --syn --dport 143 -m connlimit --connlimit-above 2 -j REJECT
Thanks,
-deano
Regards Marten
participants (5)
-
Dean Blackburn
-
Marten Lehmann
-
Peter Fern
-
tblader
-
Trent Lloyd