[Dovecot] OpenBSD and too many open files

older
Re: [Dovecot] OpenBSD and too many...

Bryan Vyhmeister

1 Aug 2007 1 Aug '07

7:19 p.m.

I posted some questions a few months back and received lots of great
help in getting my dovecot/postfix/LDAP setup working. Initially, I
ran into the pipe failed too many open files error. I am running this
setup on OpenBSD/sparc64 4.1-stable and so I put dovecot in its own
class and upped the maxfiles setting for that class. This was back
with dovecot 1.0 and everything worked fine. I just recently upgraded
to dovecot 1.0.2 from ports and now I am having the same problems
again. I moved kern.maxfiles for that class from 2048 up to 3404 and
yet I still run into the error. When I do, I have to restart dovecot
to get things running again.

Output of: su -c dovecot root -c 'ulimit -a' time(cpu-seconds) unlimited file(blocks) unlimited coredump(blocks) unlimited data(kbytes) 1048576 stack(kbytes) 8192 lockedmem(kbytes) 634482 memory(kbytes) 1898080 nofiles(descriptors) 3404 processes 1044

sysctl kern.maxfiles gives: kern.maxfiles=3404

My main IMAP accounts reside on this server and I have somewhere in
the 25000 range of stored messages. Could this be causing the issue?
I only have maybe a dozen other users using this server right now but
I am planning on putting 500 users on it soon. I added another heavy
IMAP user shortly before these problems started but I don't know if
that has anything to do with it or not. Likewise, I am not sure if
the dovecot 1.0.2 upgrade had any effect either although it did
coincide. Any ideas? Thank you.

Bryan

Show replies by date

Quentin Garnier

1 Aug 1 Aug

7:25 p.m.

On Wed, Aug 01, 2007 at 09:19:11AM -0700, Bryan Vyhmeister wrote:

...

I posted some questions a few months back and received lots of great
help in getting my dovecot/postfix/LDAP setup working. Initially, I
ran into the pipe failed too many open files error. I am running this
setup on OpenBSD/sparc64 4.1-stable and so I put dovecot in its own
class and upped the maxfiles setting for that class. This was back
with dovecot 1.0 and everything worked fine. I just recently upgraded
to dovecot 1.0.2 from ports and now I am having the same problems
again. I moved kern.maxfiles for that class from 2048 up to 3404 and
yet I still run into the error. When I do, I have to restart dovecot
to get things running again.

Sounds more like a file descriptor leak. Check fstat output so see what kind of descriptors are leaked, how fast, etc.

-- Quentin Garnier - cube@cubidou.net - cube@NetBSD.org "You could have made it, spitting out benchmarks Owe it to yourself not to fail" Amplifico, Spitting Out Benchmarks, Hometakes Vol. 2, 2005.

Bryan Vyhmeister

7:36 p.m.

On Aug 1, 2007, at 9:25 AM, Quentin Garnier wrote:

...

Sounds more like a file descriptor leak. Check fstat output so see
what kind of descriptors are leaked, how fast, etc.

Thank you for your quick response. Unfortunately, I restarted dovecot
this morning and so this probably isn't very helpful now but here it
is anyway. What should I be looking for to indicate a leak? The
outputs of 'fstat -u _dovecot' and 'fstat -u _dovecot_login' follow:

USER CMD PID FD MOUNT INUM MODE R/W DV|SZ _dovecot imap 3328 wd /d 1504391 drwx------ r 1024 _dovecot imap 3328 0* internet stream tcp 0x4000d4cba50
66.81.83.13:143 <-- 66.81.83.3:52591 _dovecot imap 3328 1* internet stream tcp 0x4000d4cba50
66.81.83.13:143 <-- 66.81.83.3:52591 _dovecot imap 3328 2 pipe 0x4000d439c90 state: _dovecot imap 3328 3 / 475071 crw-r--r-- r urandom _dovecot imap 3328 4 kqueue 0x4000d4c7d00 0 state: W _dovecot imap 3328 5 pipe 0x4000d6c7480 state: _dovecot imap 3328 6 pipe 0x4000d6c7480 state: _dovecot imap 3328 9 /d 1526304 -rw------- rw 152352 _dovecot imap 3328 10 /d 1526287 -rw------- rw 9344 _dovecot imap 3328 11 /d 1514861 -rw------- rw 7173120 _dovecot imap 10524 wd /d 1508155 drwx------ r 2048 _dovecot imap 10524 0* internet stream tcp 0x4000d4cb820
66.81.83.13:143 <-- 66.81.83.3:52699 _dovecot imap 10524 1* internet stream tcp 0x4000d4cb820
66.81.83.13:143 <-- 66.81.83.3:52699 _dovecot imap 10524 2 pipe 0x4000d6c7700 state: _dovecot imap 10524 3 / 475071 crw-r--r-- r urandom _dovecot imap 10524 4 kqueue 0x4000d4c7200 0 state: W _dovecot imap 10524 5 pipe 0x4000d6c7d00 state: _dovecot imap 10524 6 pipe 0x4000d6c7d00 state: _dovecot imap 32143 wd /d 1628354 drwx------ r 512 _dovecot imap 32143 0* internet stream tcp 0x4000d2f86c0
66.81.83.13:143 <-- 66.81.83.3:50414 _dovecot imap 32143 1* internet stream tcp 0x4000d2f86c0
66.81.83.13:143 <-- 66.81.83.3:50414 _dovecot imap 32143 2 pipe 0x4000d6c7080 state: _dovecot imap 32143 3 / 475071 crw-r--r-- r urandom _dovecot imap 32143 4 kqueue 0x4000d4c7a00 0 state: W _dovecot imap 32143 5 pipe 0x4000d439a90 state: _dovecot imap 32143 6 pipe 0x4000d439a90 state: _dovecot imap 1563 wd /d 1794316 drwx------ r 512 _dovecot imap 1563 0* internet stream tcp 0x4000d4ca010
66.81.83.13:143 <-- 10.80.1.249:24288 _dovecot imap 1563 1* internet stream tcp 0x4000d4ca010
66.81.83.13:143 <-- 10.80.1.249:24288 _dovecot imap 1563 2 pipe 0x4000d6c7a80 state: _dovecot imap 1563 3 / 475071 crw-r--r-- r urandom _dovecot imap 1563 4 kqueue 0x4000d4c6200 0 state: W _dovecot imap 1563 5 pipe 0x4000d6c7c00 state: _dovecot imap 1563 6 pipe 0x4000d6c7c00 state: _dovecot imap 9679 wd /d 1772316 drwx------ r 512 _dovecot imap 9679 0* internet stream tcp 0x4000d2f93e0
66.81.83.13:143 <-- 66.81.83.3:54735 _dovecot imap 9679 1* internet stream tcp 0x4000d2f93e0
66.81.83.13:143 <-- 66.81.83.3:54735 _dovecot imap 9679 2 pipe 0x4000d6c6e00 state: _dovecot imap 9679 3 / 475071 crw-r--r-- r urandom _dovecot imap 9679 4 kqueue 0x4000d4c7c00 0 state: W _dovecot imap 9679 5 pipe 0x4000d6c7980 state: _dovecot imap 9679 6 pipe 0x4000d6c7980 state: _dovecot imap 27728 wd /d 1772316 drwx------ r 512 _dovecot imap 27728 0* internet stream tcp 0x4000d2f8030
66.81.83.13:143 <-- 66.81.83.3:61851 _dovecot imap 27728 1* internet stream tcp 0x4000d2f8030
66.81.83.13:143 <-- 66.81.83.3:61851 _dovecot imap 27728 2 pipe 0x4000d6c7c80 state: _dovecot imap 27728 3 / 475071 crw-r--r-- r urandom _dovecot imap 27728 4 kqueue 0x4000d4c7300 0 state: W _dovecot imap 27728 5 pipe 0x4000d6c7680 state: _dovecot imap 27728 6 pipe 0x4000d6c7680 state: _dovecot imap 27728 7 /d 1772398 -rw------- rw 384 _dovecot imap 27728 8 /d 1772332 -rw------- rw 3480 _dovecot imap 27728 9 /d 1772337 -rw------- rw 21504 _dovecot imap 23686 wd /d 1525067 drwx------ r 512 _dovecot imap 23686 0* internet stream tcp 0x4000d4cbc80
66.81.83.13:143 <-- 66.81.83.3:56740 _dovecot imap 23686 1* internet stream tcp 0x4000d4cbc80
66.81.83.13:143 <-- 66.81.83.3:56740 _dovecot imap 23686 2 pipe 0x4000d439d90 state: _dovecot imap 23686 3 / 475071 crw-r--r-- r urandom _dovecot imap 23686 4 kqueue 0x4000d4c7b00 0 state: W _dovecot imap 23686 5 pipe 0x4000d438490 state: _dovecot imap 23686 6 pipe 0x4000d438490 state: _dovecot imap 18754 wd /d 1504391 drwx------ r 1024 _dovecot imap 18754 0* internet stream tcp 0x4000ce92720
66.81.83.13:143 <-- 66.81.83.3:56072 _dovecot imap 18754 1* internet stream tcp 0x4000ce92720
66.81.83.13:143 <-- 66.81.83.3:56072 _dovecot imap 18754 2 pipe 0x4000d439c10 state: _dovecot imap 18754 3 / 475071 crw-r--r-- r urandom _dovecot imap 18754 4 kqueue 0x4000d4c7e00 0 state: W _dovecot imap 18754 5 pipe 0x4000d439510 state: _dovecot imap 18754 6 pipe 0x4000d439510 state: _dovecot imap 18754 9 /d 1514852 -rw------- rw 28064 _dovecot imap 18754 10 /d 1516770 -rw------- rw 59832 _dovecot imap 18754 11 /d 1514853 -rw------- rw 1313792 _dovecot imap 25773 wd /d 1607430 drwx------ r 512 _dovecot imap 25773 0* internet stream tcp 0x4000ce93d00
66.81.83.13:143 <-- 66.81.83.3:55608 _dovecot imap 25773 1* internet stream tcp 0x4000ce93d00
66.81.83.13:143 <-- 66.81.83.3:55608 _dovecot imap 25773 2 pipe 0x4000d6c7800 state: _dovecot imap 25773 3 / 475071 crw-r--r-- r urandom _dovecot imap 25773 4 kqueue 0x4000d4c7900 0 state: W _dovecot imap 25773 5 pipe 0x4000d439910 state: _dovecot imap 25773 6 pipe 0x4000d439910 state:

fstat -u _dovecot_login

USER CMD PID FD MOUNT INUM MODE R/W DV|SZ _dovecot imap-login 3436 root / 1319570 drwxr-x--- r 512 _dovecot imap-login 3436 wd / 1319570 drwxr-x--- r 512 _dovecot imap-login 3436 0* internet stream tcp 0x4000d50fa40 *:143 _dovecot imap-login 3436 1* internet stream tcp 0x4000d4ca240 *:993 _dovecot imap-login 3436 2 pipe 0x4000d439710 state: _dovecot imap-login 3436 3* unix stream 0x400024f7c00 <->
0x4000250f400 _dovecot imap-login 3436 4 / 475071 crw-r--r-- r urandom _dovecot imap-login 3436 5 kqueue 0x4000d4c6400 0 state: W _dovecot imap-login 3436 6 pipe 0x4000d439f10 state: _dovecot imap-login 3436 7 pipe 0x4000d439f10 state: _dovecot imap-login 3436 9* unix stream 0x40002485e00 <->
0x400024fb900 _dovecot imap-login 18776 root / 1319570 drwxr-x--- r 512 _dovecot imap-login 18776 wd / 1319570 drwxr-x--- r 512 _dovecot imap-login 18776 0* internet stream tcp 0x4000d50fa40 *:143 _dovecot imap-login 18776 1* internet stream tcp 0x4000d4ca240 *:993 _dovecot imap-login 18776 2 pipe 0x4000d439110 state: _dovecot imap-login 18776 3* unix stream 0x40002515480 <->
0x40002523d80 _dovecot imap-login 18776 4 / 475071 crw-r--r-- r urandom _dovecot imap-login 18776 5 kqueue 0x4000d4c6900 0 state: W _dovecot imap-login 18776 6 pipe 0x4000d439810 state: _dovecot imap-login 18776 7 pipe 0x4000d439810 state: _dovecot imap-login 18776 9* unix stream 0x40002405e80 <->
0x40002528a00 _dovecot pop3-login 13031 root / 1319570 drwxr-x--- r 512 _dovecot pop3-login 13031 wd / 1319570 drwxr-x--- r 512 _dovecot pop3-login 13031 0* internet stream tcp 0x4000d2f8b20 *:110 _dovecot pop3-login 13031 1* internet stream tcp 0x4000d2f88f0 *:995 _dovecot pop3-login 13031 2 pipe 0x4000d439610 state: _dovecot pop3-login 13031 3* unix stream 0x40002517200 <->
0x40002404200 _dovecot pop3-login 13031 4 / 475071 crw-r--r-- r urandom _dovecot pop3-login 13031 5 kqueue 0x4000d4c7500 0 state: W _dovecot pop3-login 13031 6 pipe 0x4000d6c7d80 state: _dovecot pop3-login 13031 7 pipe 0x4000d6c7d80 state: _dovecot pop3-login 13031 9* unix stream 0x40002519f80 <->
0x40002527300 _dovecot imap-login 25333 root / 1319570 drwxr-x--- r 512 _dovecot imap-login 25333 wd / 1319570 drwxr-x--- r 512 _dovecot imap-login 25333 0* internet stream tcp 0x4000d50fa40 *:143 _dovecot imap-login 25333 1* internet stream tcp 0x4000d4ca240 *:993 _dovecot imap-login 25333 2 pipe 0x4000d6c7200 state: _dovecot imap-login 25333 3* unix stream 0x400023ab080 <->
0x40002422700 _dovecot imap-login 25333 4 / 475071 crw-r--r-- r urandom _dovecot imap-login 25333 5 kqueue 0x4000d4c6100 0 state: W _dovecot imap-login 25333 6 pipe 0x4000d6c7780 state: _dovecot imap-login 25333 7 pipe 0x4000d6c7780 state: _dovecot imap-login 25333 9* unix stream 0x4000238a900 <->
0x400024fbe00 _dovecot imap-login 22287 root / 1319570 drwxr-x--- r 512 _dovecot imap-login 22287 wd / 1319570 drwxr-x--- r 512 _dovecot imap-login 22287 0* internet stream tcp 0x4000d50fa40 *:143 _dovecot imap-login 22287 1* internet stream tcp 0x4000d4ca240 *:993 _dovecot imap-login 22287 2 pipe 0x4000d439990 state: _dovecot imap-login 22287 3* unix stream 0x4000250c080 <->
0x40002516300 _dovecot imap-login 22287 4 / 475071 crw-r--r-- r urandom _dovecot imap-login 22287 5 kqueue 0x4000d4c7800 0 state: W _dovecot imap-login 22287 6 pipe 0x4000d6c7900 state: _dovecot imap-login 22287 7 pipe 0x4000d6c7900 state: _dovecot imap-login 22287 9* unix stream 0x4000252c300 <->
0x400024f3600 _dovecot imap-login 23627 root / 1319570 drwxr-x--- r 512 _dovecot imap-login 23627 wd / 1319570 drwxr-x--- r 512 _dovecot imap-login 23627 0* internet stream tcp 0x4000d50fa40 *:143 _dovecot imap-login 23627 1* internet stream tcp 0x4000d4ca240 *:993 _dovecot imap-login 23627 2 pipe 0x4000d438a10 state: _dovecot imap-login 23627 3* unix stream 0x4000250ba80 <->
0x4000251ef00 _dovecot imap-login 23627 4 / 475071 crw-r--r-- r urandom _dovecot imap-login 23627 5 kqueue 0x4000d4c7000 0 state: W _dovecot imap-login 23627 6 pipe 0x4000d438d90 state: _dovecot imap-login 23627 7 pipe 0x4000d438d90 state: _dovecot imap-login 23627 9* unix stream 0x400024f3280 <->
0x40002504c00 _dovecot pop3-login 28303 root / 1319570 drwxr-x--- r 512 _dovecot pop3-login 28303 wd / 1319570 drwxr-x--- r 512 _dovecot pop3-login 28303 0* internet stream tcp 0x4000d2f8b20 *:110 _dovecot pop3-login 28303 1* internet stream tcp 0x4000d2f88f0 *:995 _dovecot pop3-login 28303 2 pipe 0x4000d438c10 state: _dovecot pop3-login 28303 3* unix stream 0x40002514500 <->
0x400024f6000 _dovecot pop3-login 28303 4 / 475071 crw-r--r-- r urandom _dovecot pop3-login 28303 5 kqueue 0x4000d4c6300 0 state: W _dovecot pop3-login 28303 6 pipe 0x4000d439b10 state: _dovecot pop3-login 28303 7 pipe 0x4000d439b10 state: _dovecot pop3-login 28303 9* unix stream 0x400024f7a00 <->
0x400024f3980 _dovecot pop3-login 20519 root / 1319570 drwxr-x--- r 512 _dovecot pop3-login 20519 wd / 1319570 drwxr-x--- r 512 _dovecot pop3-login 20519 0* internet stream tcp 0x4000d2f8b20 *:110 _dovecot pop3-login 20519 1* internet stream tcp 0x4000d2f88f0 *:995 _dovecot pop3-login 20519 2 pipe 0x4000d439d10 state: _dovecot pop3-login 20519 3* unix stream 0x4000252c900 <->
0x40002484880 _dovecot pop3-login 20519 4 / 475071 crw-r--r-- r urandom _dovecot pop3-login 20519 5 kqueue 0x4000d4c6f00 0 state: W _dovecot pop3-login 20519 6 pipe 0x4000d439e90 state: _dovecot pop3-login 20519 7 pipe 0x4000d439e90 state: _dovecot pop3-login 20519 9* unix stream 0x40002504580 <->
0x4000250f680 _dovecot pop3-login 13249 root / 1319570 drwxr-x--- r 512 _dovecot pop3-login 13249 wd / 1319570 drwxr-x--- r 512 _dovecot pop3-login 13249 0* internet stream tcp 0x4000d2f8b20 *:110 _dovecot pop3-login 13249 1* internet stream tcp 0x4000d2f88f0 *:995 _dovecot pop3-login 13249 2 pipe 0x4000d439490 state: _dovecot pop3-login 13249 3* unix stream 0x4000251be80 <->
0x40002372800 _dovecot pop3-login 13249 4 / 475071 crw-r--r-- r urandom _dovecot pop3-login 13249 5 kqueue 0x4000d4c6c00 0 state: W _dovecot pop3-login 13249 6 pipe 0x4000d438c90 state: _dovecot pop3-login 13249 7 pipe 0x4000d438c90 state: _dovecot pop3-login 13249 9* unix stream 0x4000252b080 <->
0x4000250fa00 _dovecot pop3-login 30768 root / 1319570 drwxr-x--- r 512 _dovecot pop3-login 30768 wd / 1319570 drwxr-x--- r 512 _dovecot pop3-login 30768 0* internet stream tcp 0x4000d2f8b20 *:110 _dovecot pop3-login 30768 1* internet stream tcp 0x4000d2f88f0 *:995 _dovecot pop3-login 30768 2 pipe 0x4000d439b90 state: _dovecot pop3-login 30768 3* unix stream 0x40002520600 <->
0x40002484c00 _dovecot pop3-login 30768 4 / 475071 crw-r--r-- r urandom _dovecot pop3-login 30768 5 kqueue 0x4000d4c6d00 0 state: W _dovecot pop3-login 30768 6 pipe 0x4000d439410 state: _dovecot pop3-login 30768 7 pipe 0x4000d439410 state: _dovecot pop3-login 30768 9* unix stream 0x40002423880 <->
0x400023bd700

Timo Sirainen

7:41 p.m.

On 1.8.2007, at 19.36, Bryan Vyhmeister wrote:

...

On Aug 1, 2007, at 9:25 AM, Quentin Garnier wrote:

...
Sounds more like a file descriptor leak. Check fstat output so
see what kind of descriptors are leaked, how fast, etc.

Thank you for your quick response. Unfortunately, I restarted
dovecot this morning and so this probably isn't very helpful now
but here it is anyway. What should I be looking for to indicate a
leak? The outputs of 'fstat -u _dovecot' and 'fstat -u
_dovecot_login' follow:

You didn't say which process got the "too many files" error, but
usually it's the Dovecot master process. Could you show the exact
error message so I can be sure? Here you showed only imap, imap-login
and pop3-login processes, not dovecot process.

Bryan Vyhmeister

7:51 p.m.

On Aug 1, 2007, at 9:41 AM, Timo Sirainen wrote:

...

You didn't say which process got the "too many files" error, but
usually it's the Dovecot master process. Could you show the exact
error message so I can be sure? Here you showed only imap, imap- login and pop3-login processes, not dovecot process.

You're right. Sorry, I couldn't find the error message initially but
I just located it. It appears like this particular error appears in
the context of a dictionary attack.

Jul 30 07:31:02 a dovecot: pop3-login: Aborted login: user=<aaron>,
method=PLAIN, rip=69.3.64.3, lip=66.81.83.13 Jul 30 07:31:02 a dovecot: pop3-login: Aborted login: user=<support>,
method=PLAIN, rip=69.3.64.3, lip=66.81.83.13 Jul 30 07:31:02 a dovecot: pop3-login: Aborted login: user=<support>,
method=PLAIN, rip=69.3.64.3, lip=66.81.83.13 Jul 30 07:31:02 a dovecot: pop3-login: Aborted login: user=<aaron>,
method=PLAIN, rip=69.3.64.3, lip=66.81.83.13 Jul 30 07:31:02 a dovecot: pop3-login: Aborted login: user=<aaron>,
method=PLAIN, rip=69.3.64.3, lip=66.81.83.13 Jul 30 07:31:03 a dovecot: pipe() failed: Too many open files Jul 30 07:31:03 a dovecot: Temporary failure in creating login
processes, slowing down for now Jul 30 07:31:03 a dovecot: Created login processes successfully,
unstalling Jul 30 07:31:04 a dovecot: pop3-login: Aborted login: user=<spam>,
method=PLAIN, rip=69.3.64.3, lip=66.81.83.13 Jul 30 07:31:04 a dovecot: pop3-login: Aborted login: user=<abigail>,
method=PLAIN, rip=69.3.64.3, lip=66.81.83.13 Jul 30 07:31:04 a dovecot: pop3-login: Aborted login: user=<abraham>,
method=PLAIN, rip=69.3.64.3, lip=66.81.83.13

Jul 28 21:13:01 a dovecot: pipe() failed: Too many open files Jul 28 21:13:55 a postfix/smtpd[14992]: connect from
exprod6mx146.postini.com[64.18.1.118] Jul 28 21:13:55 a postfix/smtpd[14992]: F0A36142011:
client=exprod6mx146.postini.com[64.18.1.118] Jul 28 21:13:56 a postfix/smtpd[14992]: disconnect from
exprod6mx146.postini.com[64.18.1.118] Jul 28 21:14:01 a dovecot: pipe() failed: Too many open files Jul 28 21:15:01 a dovecot: pipe() failed: Too many open files Jul 28 21:15:10 a postfix/smtpd[14992]: connect from
exprod6mx183.postini.com[64.18.1.74] Jul 28 21:15:10 a postfix/smtpd[14992]: D28A8142011:
client=exprod6mx183.postini.com[64.18.1.74] Jul 28 21:15:10 a postfix/smtpd[14992]: disconnect from
exprod6mx183.postini.com[64.18.1.74]

Also, here is fstat output for the master process:

USER CMD PID FD MOUNT INUM MODE R/W DV|SZ root dovecot 5846 wd / 1319569 drwxr-xr-x r 512 root dovecot 5846 0 / 475354 crw-rw-rw- r null root dovecot 5846 1 / 475354 crw-rw-rw- r null root dovecot 5846 2 / 475354 crw-rw-rw- r null root dovecot 5846 3 / 475354 crw-rw-rw- r null root dovecot 5846 4 / 475354 crw-rw-rw- r null root dovecot 5846 5* internet stream tcp 0x4000d50fa40 *:143 root dovecot 5846 6* internet stream tcp 0x4000d4ca240 *:993 root dovecot 5846 7* internet stream tcp 0x4000d2f8b20 *:110 root dovecot 5846 8* internet stream tcp 0x4000d2f88f0 *:995 root dovecot 5846 9 kqueue 0x4000d4c7600 0 state: W root dovecot 5846 10* unix dgram 0x4000252c400 <->
0x400022ac580 root dovecot 5846 11 pipe 0x4000d438f10 state: root dovecot 5846 12 pipe 0x4000d438f10 state: root dovecot 5846 13* unix stream 0x4000252bf80 root dovecot 5846 14* unix stream 0x4000238a500 root dovecot 5846 15* unix stream 0x4000251c680 <->
0x4000250d000 root dovecot 5846 16* unix stream 0x40002404200 <->
0x40002517200 root dovecot 5846 17 pipe 0x4000d439310 state: root dovecot 5846 18* unix stream 0x40002484c00 <->
0x40002520600 root dovecot 5846 19* unix stream 0x400023ab280 root dovecot 5846 20* unix stream 0x40002501a00 <->
0x40002501000 root dovecot 5846 21* unix stream 0x40002372800 <->
0x4000251be80 root dovecot 5846 22 pipe 0x4000d439b90 state: root dovecot 5846 23* unix stream 0x40002484880 <->
0x4000252c900 root dovecot 5846 24 pipe 0x4000d439490 state: root dovecot 5846 25* unix stream 0x400024f6000 <->
0x40002514500 root dovecot 5846 26 pipe 0x4000d439d10 state: root dovecot 5846 27* unix stream 0x4000250bb00 <->
0x400024f6280 root dovecot 5846 28 pipe 0x4000d438c10 state: root dovecot 5846 30 pipe 0x4000d439a90 state: root dovecot 5846 31 pipe 0x4000d439c10 state: root dovecot 5846 32 pipe 0x4000d439790 state: root dovecot 5846 33 pipe 0x4000d6c7700 state: root dovecot 5846 34 pipe 0x4000d439610 state: root dovecot 5846 35* unix stream 0x4000251ef00 <->
0x4000250ba80 root dovecot 5846 37* unix stream 0x40002516300 <->
0x4000250c080 root dovecot 5846 38 pipe 0x4000d438a10 state: root dovecot 5846 39 pipe 0x4000d6c7400 state: root dovecot 5846 40 pipe 0x4000d6c6e00 state: root dovecot 5846 41 pipe 0x4000d439910 state: root dovecot 5846 42 pipe 0x4000d439c90 state: root dovecot 5846 43 pipe 0x4000d439990 state: root dovecot 5846 44 pipe 0x4000d6c7c80 state: root dovecot 5846 45 pipe 0x4000d438a90 state: root dovecot 5846 46 pipe 0x4000d6c7a80 state: root dovecot 5846 47 pipe 0x4000d6c7280 state: root dovecot 5846 50 pipe 0x4000d6c6f00 state: root dovecot 5846 53* unix stream 0x40002423a80 <->
0x4000233a500 root dovecot 5846 57 pipe 0x4000d439a10 state:

Quentin Garnier

7:43 p.m.

On Wed, Aug 01, 2007 at 09:36:49AM -0700, Bryan Vyhmeister wrote:

...

On Aug 1, 2007, at 9:25 AM, Quentin Garnier wrote:

...
Sounds more like a file descriptor leak. Check fstat output so see
what kind of descriptors are leaked, how fast, etc.

Thank you for your quick response. Unfortunately, I restarted dovecot
this morning and so this probably isn't very helpful now but here it
is anyway. What should I be looking for to indicate a leak? The
outputs of 'fstat -u _dovecot' and 'fstat -u _dovecot_login' follow:

Oh, you're using kqueue. I didn't know OpenBSD eventually grew support for it.

FWIW, I've stopped using dovecot's kqueue code quite some time ago because it would leak one fd almost every login, leading to the kind of situation you experience. I wish I had the time to debug that, but unfortunately...

-- Quentin Garnier - cube@cubidou.net - cube@NetBSD.org "You could have made it, spitting out benchmarks Owe it to yourself not to fail" Amplifico, Spitting Out Benchmarks, Hometakes Vol. 2, 2005.

Timo Sirainen

7:52 p.m.

On 1.8.2007, at 19.43, Quentin Garnier wrote:

...

FWIW, I've stopped using dovecot's kqueue code quite some time ago because it would leak one fd almost every login, leading to the
kind of situation you experience. I wish I had the time to debug that, but unfortunately...

I thought this happened only with older NetBSDs. I tested it myself
once with 3.0 (or 3.1?) and I couldn't get it to leak.

But yes, with some NetBSD versions it does leak one pipe per each
died process because kqueue doesn't notice that the pipe was closed
by the other side and give an event, or something like that.

Bryan Vyhmeister

7:56 p.m.

On Aug 1, 2007, at 9:52 AM, Timo Sirainen wrote:

...

On 1.8.2007, at 19.43, Quentin Garnier wrote:

...
FWIW, I've stopped using dovecot's kqueue code quite some time ago because it would leak one fd almost every login, leading to the
kind of situation you experience. I wish I had the time to debug that, but unfortunately...

I thought this happened only with older NetBSDs. I tested it myself
once with 3.0 (or 3.1?) and I couldn't get it to leak.

But yes, with some NetBSD versions it does leak one pipe per each
died process because kqueue doesn't notice that the pipe was closed
by the other side and give an event, or something like that.

I am guessing that OpenBSD is affected in the same way?

Bryan

Quentin Garnier

8:01 p.m.

On Wed, Aug 01, 2007 at 07:52:33PM +0300, Timo Sirainen wrote:

...

On 1.8.2007, at 19.43, Quentin Garnier wrote:

...
FWIW, I've stopped using dovecot's kqueue code quite some time ago because it would leak one fd almost every login, leading to the
kind of situation you experience. I wish I had the time to debug that, but unfortunately...

I thought this happened only with older NetBSDs. I tested it myself
once with 3.0 (or 3.1?) and I couldn't get it to leak.

Yes, older NetBSDs. What do you think OpenBSD is?

Nah, just kidding. I don't remember what tests I did at the time. I do remember that I couldn't reproduce it as easily as I wanted to, and the likely cause was that I was doing my tests on a kernel more recent than the 2.0 of my prod servers.

...

But yes, with some NetBSD versions it does leak one pipe per each
died process because kqueue doesn't notice that the pipe was closed
by the other side and give an event, or something like that.

Which does look like what is experienced here. Now, I don't know if we can locate easily what fix needs to be applied to OpenBSD.

-- Quentin Garnier - cube@cubidou.net - cube@NetBSD.org "You could have made it, spitting out benchmarks Owe it to yourself not to fail" Amplifico, Spitting Out Benchmarks, Hometakes Vol. 2, 2005.

Quentin Garnier

8:37 p.m.

On Wed, Aug 01, 2007 at 07:01:35PM +0200, Quentin Garnier wrote: [...]

...

Which does look like what is experienced here. Now, I don't know if we can locate easily what fix needs to be applied to OpenBSD.

Check if this applies to OpenBSD:

http://cvsweb.netbsd.org/bsdweb.cgi/src/sys/kern/sys_pipe.c.diff?r1=1.65&r2=1.66&f=h

-- Quentin Garnier - cube@cubidou.net - cube@NetBSD.org "You could have made it, spitting out benchmarks Owe it to yourself not to fail" Amplifico, Spitting Out Benchmarks, Hometakes Vol. 2, 2005.

Bryan Vyhmeister

8:56 p.m.

On Aug 1, 2007, at 10:37 AM, Quentin Garnier wrote:

...

On Wed, Aug 01, 2007 at 07:01:35PM +0200, Quentin Garnier wrote: [...]

...
Which does look like what is experienced here. Now, I don't know
if we can locate easily what fix needs to be applied to OpenBSD.

Check if this applies to OpenBSD:

http://cvsweb.netbsd.org/bsdweb.cgi/src/sys/kern/sys_pipe.c.diff? r1=1.65&r2=1.66&f=h

From looking at:

http://www.openbsd.org/cgi-bin/cvsweb/~checkout~/src/sys/kern/ sys_pipe.c?rev=1.51&content-type=text/plain

I can't quite tell. There are significant differences. I'll have to
take a closer look after a work engagement.

Bryan

Quentin Garnier

9:09 p.m.

On Wed, Aug 01, 2007 at 10:56:26AM -0700, Bryan Vyhmeister wrote:

...

On Aug 1, 2007, at 10:37 AM, Quentin Garnier wrote:

...
On Wed, Aug 01, 2007 at 07:01:35PM +0200, Quentin Garnier wrote: [...]

...
Which does look like what is experienced here. Now, I don't know
if we can locate easily what fix needs to be applied to OpenBSD.

Check if this applies to OpenBSD:

http://cvsweb.netbsd.org/bsdweb.cgi/src/sys/kern/sys_pipe.c.diff? r1=1.65&r2=1.66&f=h

From looking at:

http://www.openbsd.org/cgi-bin/cvsweb/~checkout~/src/sys/kern/ sys_pipe.c?rev=1.51&content-type=text/plain

I can't quite tell. There are significant differences. I'll have to
take a closer look after a work engagement.

Well, the problem is Christos did some other stuff in that commit than just fixing the bug. Try the attached patch.

-- Quentin Garnier - cube@cubidou.net - cube@NetBSD.org "You could have made it, spitting out benchmarks Owe it to yourself not to fail" Amplifico, Spitting Out Benchmarks, Hometakes Vol. 2, 2005.

Bryan Vyhmeister

2 Aug 2 Aug

2:22 a.m.

On Aug 1, 2007, at 11:09 AM, Quentin Garnier wrote:

...

Well, the problem is Christos did some other stuff in that commit than just fixing the bug. Try the attached patch.

Thank you for the patch. It applied cleanly and I have a kernel that
reflects the patch running right now. So far so good. I will report
back if anything unusual happens. Otherwise, I will report back first
thing next week if everything works fine from here on out. Thank you
all for your help.

Bryan

Bryan Vyhmeister

7 Aug 7 Aug

8:05 p.m.

On Aug 1, 2007, at 4:22 PM, Bryan Vyhmeister wrote:

...

On Aug 1, 2007, at 11:09 AM, Quentin Garnier wrote:

...
Well, the problem is Christos did some other stuff in that commit
than just fixing the bug. Try the attached patch.

Thank you for the patch. It applied cleanly and I have a kernel
that reflects the patch running right now. So far so good. I will
report back if anything unusual happens. Otherwise, I will report
back first thing next week if everything works fine from here on
out. Thank you all for your help.

Well, I have not had a single instance of the "Too many files" error
since. Everything is working smoothly. I'll see about getting the
patch into OpenBSD if possible. I'll get that tutorial done as well.

Bryan

Brad

9:15 p.m.

On Tue, 7 Aug 2007 10:05:32 -0700 Bryan Vyhmeister dovecot@bsdjournal.net wrote:

...

On Aug 1, 2007, at 4:22 PM, Bryan Vyhmeister wrote:

...
On Aug 1, 2007, at 11:09 AM, Quentin Garnier wrote:

...
Well, the problem is Christos did some other stuff in that commit
than just fixing the bug. Try the attached patch.

Thank you for the patch. It applied cleanly and I have a kernel
that reflects the patch running right now. So far so good. I will
report back if anything unusual happens. Otherwise, I will report
back first thing next week if everything works fine from here on
out. Thank you all for your help.

Well, I have not had a single instance of the "Too many files" error
since. Everything is working smoothly. I'll see about getting the
patch into OpenBSD if possible. I'll get that tutorial done as well.

Bryan

I already passed this around for review and it has been commited.

http://marc.info/?l=openbsd-cvs&m=118648637312063&w=2

Bryan Vyhmeister

9:24 p.m.

On Aug 7, 2007, at 11:15 AM, Brad wrote:

...

I already passed this around for review and it has been commited.

http://marc.info/?l=openbsd-cvs&m=118648637312063&w=2

Great! Thank you very much.

Bryan

6271

Age (days ago)

6277

Last active (days ago)

List overview

15 comments

4 participants

participants (4)

Brad
Bryan Vyhmeister
Quentin Garnier
Timo Sirainen