[Dovecot] BSD people, please test kqueue changes
I rewrote much of the kqueue code since the old code didn't handle properly the case when both input and output I/O handlers were added to the same file descriptor with different callbacks. This is done commonly in Dovecot, so I'm wondering why more people didn't complain about problems with it.. :)
I don't anyway have access to any BSDs so the code is completely untested. Please test how it works before 1.0rc7 :)
http://dovecot.org/nightly/dovecot-latest.tar.gz
I'd guess the hangs caused by recent RC versions were because the default was to use notify=kqueue instead of notify=none. I changed this back to none unless --with-ioloop=kqueue is used.
There are a couple of things in the code that I'm not completely sure of. The kqueue() manual page says that EV_ADD "modifies" the event if it already exists. What exactly does that mean? Does it mean that the filter is replaced, or that it's ORed into the existing one? The same question for the EV_DELETE, does it delete everything or only those filters given in parameters?
The code currently assumes ORing and EV_DELETE deleting only the given filters, since this is what the original code also did.
I suppose the data and udata parameters don't matter in EV_DELETE since they weren't given originally in ioloop-notify-kqueue.c either, so I left them that way.
configure resulted in:
I/O loop method ..................... : poll File change notification method ..... : none
Where 1.0.rc6 did:
I/O loop method ..................... : poll File change notification method ..... : kqueue
What configuration options do you want used?
(I am using FreeBSD 4-something.)
On Wed, 2006-08-16 at 11:35 -0500, Jeremy C. Reed wrote:
configure resulted in:
I/O loop method ..................... : poll File change notification method ..... : none
Yes, that's the correct default.
What configuration options do you want used?
--with-ioloop=kqueue should set both ioloop and notify to kqueue.
Hi,
On Wed, 16 Aug 2006, Timo Sirainen wrote:
On Wed, 2006-08-16 at 11:35 -0500, Jeremy C. Reed wrote:
configure resulted in:
I/O loop method ..................... : poll File change notification method ..... : none
Yes, that's the correct default.
What configuration options do you want used?
--with-ioloop=kqueue should set both ioloop and notify to kqueue.
this is on FreeBSD 6.1. More precisely RELENG_6 from end of june 2006.
---snipp---
if gcc -DHAVE_CONFIG_H -I. -I. -I../.. -std=gnu99 -g -O2 -Wall -W -Wmissing-prototypes -Wmissing-declarations -Wpointer-arith -Wchar-subscripts -Wformat=2 -Wbad-function-cast -MT ioloop-notify-kqueue.o -MD -MP -MF ".deps/ioloop-notify-kqueue.Tpo" -c -o ioloop-notify-kqueue.o ioloop-notify-kqueue.c; then mv -f ".deps/ioloop-notify-kqueue.Tpo" ".deps/ioloop-notify-kqueue.Po"; else rm -f ".deps/ioloop-notify-kqueue.Tpo"; exit 1; fi
ioloop-notify-kqueue.c: In function event_callback': ioloop-notify-kqueue.c:45: error: ret' undeclared (first use in this function)
ioloop-notify-kqueue.c:45: error: (Each undeclared identifier is reported only once
ioloop-notify-kqueue.c:45: error: for each function it appears in.)
ioloop-notify-kqueue.c: In function io_loop_notify_add': ioloop-notify-kqueue.c:113: error: incompatible types in initialization ioloop-notify-kqueue.c: In function io_loop_notify_remove':
ioloop-notify-kqueue.c:138: error: incompatible types in initialization
*** Error code 1
Stop in /home/ck/src/dovecot/dovecot-1.0.rc6/src/lib. *** Error code 1
Stop in /home/ck/src/dovecot/dovecot-1.0.rc6/src. *** Error code 1
Stop in /home/ck/src/dovecot/dovecot-1.0.rc6. *** Error code 1
Stop in /home/ck/src/dovecot/dovecot-1.0.rc6. ---snipp---
Greetings Christian
-- Christian Kratzer ck@cksoft.de CK Software GmbH http://www.cksoft.de/ Phone: +49 7452 889 135 Fax: +49 7452 889 136
What configuration options do you want used?
--with-ioloop=kqueue should set both ioloop and notify to kqueue.
Building fails. First: --- src/lib/ioloop-notify-kqueue.c.orig Wed Aug 16 10:50:24 2006 +++ src/lib/ioloop-notify-kqueue.c Wed Aug 16 10:51:31 2006 @@ -34,6 +34,7 @@ struct io *io; struct kevent ev; struct timespec ts; + int ret; if (gettimeofday(&ioloop_timeval, &ioloop_timezone) < 0) i_fatal("gettimeofday() failed: %m"); But still fails with: source='ioloop-notify-kqueue.c' object='ioloop-notify-kqueue.o' libtool=no DEPDIR=.deps depmode=gcc /bin/bash ../../depcomp gcc -DHAVE_CONFIG_H -I. -I. -I../.. -g -O2 -Wall -W -Wmissing-prototypes -Wmissing-declarations -Wpointer-arith -Wchar-subscripts -Wformat=2 -Wbad-function-cast -c ioloop-notify-kqueue.c ioloop-notify-kqueue.c: In function `io_loop_notify_add': ioloop-notify-kqueue.c:114: invalid type argument of `->' ioloop-notify-kqueue.c:114: invalid type argument of `->' ioloop-notify-kqueue.c:114: invalid type argument of `->' ioloop-notify-kqueue.c:114: invalid type argument of `->' ioloop-notify-kqueue.c:115: invalid type argument of `->' ioloop-notify-kqueue.c:115: invalid type argument of `->' ioloop-notify-kqueue.c: In function `io_loop_notify_remove': ioloop-notify-kqueue.c:139: invalid type argument of `->' ioloop-notify-kqueue.c:139: invalid type argument of `->' ioloop-notify-kqueue.c:139: invalid type argument of `->' ioloop-notify-kqueue.c:139: invalid type argument of `->' ioloop-notify-kqueue.c:139: invalid type argument of `->' ioloop-notify-kqueue.c:139: invalid type argument of `->' *** Error code 1
Timo Sirainen wrote:
I get the following compile error:
ioloop-notify-kqueue.c: In function event_callback': ioloop-notify-kqueue.c:45: error: ret' undeclared (first use in this
function)
ioloop-notify-kqueue.c:45: error: (Each undeclared identifier is
reported only once
ioloop-notify-kqueue.c:45: error: for each function it appears in.)
ioloop-notify-kqueue.c: In function io_loop_notify_add': ioloop-notify-kqueue.c:113: error: incompatible types in initialization ioloop-notify-kqueue.c: In function io_loop_notify_remove':
ioloop-notify-kqueue.c:138: error: incompatible types in initialization
declaring an int ret fixed the first error. I'm not sure about the latter two.
my configure command: ./configure --without-shadow --localstatedir=/var --with-ioloop=kqueue --with-cyrus-sasl2 --without-gssapi --without-vpopmail --prefix=/usr/local --build=i386-portbld-freebsd5.3
I'm on FreeBSD 5.3.
Thanks, Todd
On Wed, 2006-08-16 at 19:17 +0300, Timo Sirainen wrote:
I don't anyway have access to any BSDs so the code is completely untested. Please test how it works before 1.0rc7 :)
OK, so that didn't even compile. Updated it, how about now?
Hi,
On Wed, 16 Aug 2006, Timo Sirainen wrote:
On Wed, 2006-08-16 at 19:17 +0300, Timo Sirainen wrote:
I don't anyway have access to any BSDs so the code is completely untested. Please test how it works before 1.0rc7 :)
OK, so that didn't even compile. Updated it, how about now?
ok it compiles now on freebsd 6.1.
Not sure if I can get around to fully testing it. Automated tests would help of course. This kind of stuff really needs hard testing to see if its stable.
Greetings Christian
-- Christian Kratzer ck@cksoft.de CK Software GmbH http://www.cksoft.de/ Phone: +49 7452 889 135 Fax: +49 7452 889 136
Timo Sirainen wrote:
On Wed, 2006-08-16 at 19:17 +0300, Timo Sirainen wrote:
I don't anyway have access to any BSDs so the code is completely untested. Please test how it works before 1.0rc7 :)
OK, so that didn't even compile. Updated it, how about now?
Just compiled on OpenBSD 3.9 (stable).
./configure --with-ioloop=kqueue --sysconfdir=/etc --localstatedir=/var --without-vpopmail --without-shadow --without-pam
So far seems to be working ok (checked with 1.5.0.5 in imap and pop3 , in TLS mode, with cert. auth). Nothing suspicious happening or in logs, as yet.
On Wed, Aug 16, 2006 at 09:09:55PM +0300, Timo Sirainen wrote:
On Wed, 2006-08-16 at 19:17 +0300, Timo Sirainen wrote:
I don't anyway have access to any BSDs so the code is completely untested. Please test how it works before 1.0rc7 :)
OK, so that didn't even compile. Updated it, how about now?
It compiles cleanly on NetBSD 3.x (both with the defaults and with kqueue). I did not yet test it any further though.
GeertOn Wed, Aug 16, 2006 at 10:47:08PM +0200, Geert Hendrickx wrote:
On Wed, Aug 16, 2006 at 09:09:55PM +0300, Timo Sirainen wrote:
OK, so that didn't even compile. Updated it, how about now?
It compiles cleanly on NetBSD 3.x (both with the defaults and with kqueue). I did not yet test it any further though.
When compiled with kqueue (as I did before), dovecot dies immediatly after startup:
Aug 16 23:06:02 dovecot: Dovecot v1.0.rc6 starting up Aug 16 23:06:03 dovecot: Unrecognized event: kevent {.ident = 16, .filter = 0x0000, .flags = 0x0001, .fflags = 0x00000000, .data = 0x00000038}, io filter = 0
GeertOn Wed, 2006-08-16 at 23:08 +0200, Geert Hendrickx wrote:
On Wed, Aug 16, 2006 at 10:47:08PM +0200, Geert Hendrickx wrote:
On Wed, Aug 16, 2006 at 09:09:55PM +0300, Timo Sirainen wrote:
OK, so that didn't even compile. Updated it, how about now?
It compiles cleanly on NetBSD 3.x (both with the defaults and with kqueue). I did not yet test it any further though.
When compiled with kqueue (as I did before), dovecot dies immediatly after startup:
Aug 16 23:06:02 dovecot: Dovecot v1.0.rc6 starting up Aug 16 23:06:03 dovecot: Unrecognized event: kevent {.ident = 16, .filter = 0x0000, .flags = 0x0001, .fflags = 0x00000000, .data = 0x00000038}, io filter = 0
Hmm.. Wonder if this is a valid event actually. Maybe the whole call should be removed, since it's possible that this happens if an I/O handler calls io_remove() for a fd that's already in the returned events list. I couldn't reproduce this doing the same with epoll code though.
So.. What if you just comment out the i_panic() call in the end of ioloop-kqueue.c, does it work then?
On Thu, Aug 17, 2006 at 12:45:46AM +0300, Timo Sirainen wrote:
Hmm.. Wonder if this is a valid event actually. Maybe the whole call should be removed, since it's possible that this happens if an I/O handler calls io_remove() for a fd that's already in the returned events list. I couldn't reproduce this doing the same with epoll code though.
So.. What if you just comment out the i_panic() call in the end of ioloop-kqueue.c, does it work then?
Yes, then of course it keeps running (in a non-production environment).
GeertOn Thu, 2006-08-17 at 08:36 +0200, Geert Hendrickx wrote:
On Thu, Aug 17, 2006 at 12:45:46AM +0300, Timo Sirainen wrote:
Hmm.. Wonder if this is a valid event actually. Maybe the whole call should be removed, since it's possible that this happens if an I/O handler calls io_remove() for a fd that's already in the returned events list. I couldn't reproduce this doing the same with epoll code though.
So.. What if you just comment out the i_panic() call in the end of ioloop-kqueue.c, does it work then?
Yes, then of course it keeps running (in a non-production environment).
I think it might just as well have not worked at all then anyway :)
Also if you now trace all of Dovecot's processes, do they show that they're nicely waiting in kevent() syscall, or is it doing lots of kevent() calls constantly? I guess that should also increase its CPU usage.
On Thu, Aug 17, 2006 at 01:21:14PM +0300, Timo Sirainen wrote:
Also if you now trace all of Dovecot's processes, do they show that they're nicely waiting in kevent() syscall, or is it doing lots of kevent() calls constantly? I guess that should also increase its CPU usage.
ktruss(1) shows this for every dovecot process around once every second:
24458 imap-login kevent(0x5, 0, 0, 0x807e000, 0x80, 0xbfbfec48) = 0 24458 imap-login gettimeofday(0x8062924, 0x806292c) = 0 24458 imap-login gettimeofday(0xbfbfec50, 0) = 0
GeertOn Thu, Aug 17, 2006 at 01:21:14PM +0300, Timo Sirainen wrote:
I think it might just as well have not worked at all then anyway :)
Sorry, I made a mistake; I removed the i_panic() call from the 1.0rc6 sources instead of the -latest ones. With i_panic() commented out in 1.0rc6, it keeps working, but in dovecot-latest, it hangs at login.
GeertTimo Sirainen wrote, On 16.8.2006 18:17:
I rewrote much of the kqueue code since the old code didn't handle properly the case when both input and output I/O handlers were added to the same file descriptor with different callbacks. This is done commonly in Dovecot, so I'm wondering why more people didn't complain about problems with it.. :)
I don't anyway have access to any BSDs so the code is completely untested. Please test how it works before 1.0rc7 :)
http://dovecot.org/nightly/dovecot-latest.tar.gz
I'd guess the hangs caused by recent RC versions were because the default was to use notify=kqueue instead of notify=none. I changed this back to none unless --with-ioloop=kqueue is used. Just a first thought before I take a closer look: The notify stuff uses kqueue but its functionality is not dependent on ioloop kqueue. It should be possible to use kqueue notify with select or poll based ioloop.
There are a couple of things in the code that I'm not completely sure of. The kqueue() manual page says that EV_ADD "modifies" the event if it already exists. What exactly does that mean? Does it mean that the filter is replaced, or that it's ORed into the existing one? The same question for the EV_DELETE, does it delete everything or only those filters given in parameters?
The code currently assumes ORing and EV_DELETE deleting only the given filters, since this is what the original code also did.
I suppose the data and udata parameters don't matter in EV_DELETE since they weren't given originally in ioloop-notify-kqueue.c either, so I left them that way.
-- VH
On Wed, 2006-08-16 at 23:21 +0200, Václav Haisman wrote:
I'd guess the hangs caused by recent RC versions were because the default was to use notify=kqueue instead of notify=none. I changed this back to none unless --with-ioloop=kqueue is used. Just a first thought before I take a closer look: The notify stuff uses kqueue but its functionality is not dependent on ioloop kqueue. It should be possible to use kqueue notify with select or poll based ioloop.
Yes, but since it seems that poll/kqueue combination started causing people to report hangs I think it's safer to just not do it by default. :)
Running it for a few hours on FreeBSD 4.x and built with --with-ioloop=kqueue
I don't see any stale "imap" processes yet and it has had a little use. I should know by tomorrow.
I've been using this for about an hour.
No stale imap processes yet but I have this in my log...
Aug 16 17:56:40 loveturtle dovecot: imap-login: Login: user=<turtle>, method=plain, rip=216.89.228.115, lip=216.91.90.11, TLS Aug 16 17:56:45 loveturtle last message repeated 3 times Aug 16 17:56:45 loveturtle dovecot: imap-login: kevent(EV_DELETE, 9) failed: No such file or directory Aug 16 17:56:45 loveturtle last message repeated 9 times Aug 16 17:56:45 loveturtle dovecot: imap-login: Sending log messages too fast, throttling.. Aug 16 17:56:45 loveturtle dovecot: imap-login: kevent(EV_DELETE, 9) failed: No such file or directory Aug 16 17:56:46 loveturtle last message repeated 5 times
No idea what that means but I thought I'd share. :-D
This is on FreeBSD/amd64 6.1-STABLE
On Wed, 16 Aug 2006, Timo Sirainen wrote:
of. The kqueue() manual page says that EV_ADD "modifies" the event if it already exists. What exactly does that mean? Does it mean that the filter is replaced, or that it's ORed into the existing one? The same
IIUC the filter fields are updated to match the new values. i.e. the later event effectively replaces the earlier event.
question for the EV_DELETE, does it delete everything or only those filters given in parameters?
Again, IIUC, only the event that EV_DELETE is set upon. Recall taht an event is identified by a filter, ident pair.
The code currently assumes ORing and EV_DELETE deleting only the given filters, since this is what the original code also did.
I don't think the ORing is correct. I can check if you like.
Andrew
participants (9)
-
Andrew
-
Christian Kratzer
-
Dillon
-
Geert Hendrickx
-
Herr Nagengast
-
Jeremy C. Reed
-
Michal Soltys
-
Timo Sirainen
-
Václav Haisman