[Dovecot] Problem with dovecot + imap idle + qmail-local delivery
Hi,
I've recently switched from courier-imap to dovecot. Everything seemed to work fine, but after I while I noticed that dovecot is occasionaly causing problems with delivery of new incoming mails.
System: Gentoo Linux, kernel 2.6.19, qmail 1.03, vpopmail 5.4.26 , dovecot 1.1.11, maildir, mail store on ext3.
Symptoms: qmail is having trouble with the mail delivery.
First I thought it had something to do with vpopmail or dovecot's quota support, but I tracked it down to qmail-local. I tried two different versions, one with Bill Shupp's maildir++ patch and one without.
With the patch, I'm getting "User_over_quota._(#5.1.1)" errors. Mail is delivered once and sender is sent a notification about the exceeded quota. Without the patch, I get "Temporary_error_on_maildir_delivery._(#4.3.0)". Mail is delivered, but delivery is tried again and again.
This only happens if a MUA is logged in to the mailbox and IMAP IDLE is being used. If the MUA is disconnected: fine. If dovecot is stopped: fine.
I tried a couple of different things, all without success:
- upgraded dovecot from 1.1.7 to 1.1.11
- compiled a new kernel with inotify support and recompiled dovecot
- migrated mail store from reiserfs to ext3
- disabling quota support in dovecot
- tried dovecot's various file locking mechanisms
Right now I'm working around this by removing IDLE from the capabilities. This solves the problem, but leaves me with a less functional IMAP server.
Any help is greatly appreciated.
Jochen
dovecot -n:
protocols: imaps imap pop3s pop3 listen(default): *:143 listen(imap): *:143 listen(pop3): *:110 ssl_listen(default): *:993 ssl_listen(imap): *:993 ssl_listen(pop3): *:995 ssl_cert_file: /etc/ssl/dovecot/server.pem ssl_key_file: /etc/ssl/dovecot/server.key disable_plaintext_auth: no verbose_ssl: yes login_dir: /var/run/dovecot/login login_executable(default): /usr/libexec/dovecot/imap-login login_executable(imap): /usr/libexec/dovecot/imap-login login_executable(pop3): /usr/libexec/dovecot/pop3-login first_valid_uid: 89 last_valid_uid: 89 first_valid_gid: 89 last_valid_gid: 89 mail_uid: 89 mail_gid: 89 mail_location: maildir:/var/vpopmail/domains/%d/%n/.maildir maildir_copy_preserve_filename: yes mail_executable(default): /usr/libexec/dovecot/imap mail_executable(imap): /usr/libexec/dovecot/imap mail_executable(pop3): /usr/libexec/dovecot/pop3 mail_plugins(default): quota imap_quota mail_plugins(imap): quota imap_quota mail_plugins(pop3): quota mail_plugin_dir(default): /usr/lib/dovecot/imap mail_plugin_dir(imap): /usr/lib/dovecot/imap mail_plugin_dir(pop3): /usr/lib/dovecot/pop3 imap_capability(default): IMAP4rev1 SASL-IR SORT THREAD=REFERENCES MULTIAPPEND UNSELECT LITERAL+ CHILDREN NAMESPACE LOGIN-REFERRALS UIDPLUS LIST-EXTENDED I18NLEVEL=1 QUOTA imap_capability(imap): IMAP4rev1 SASL-IR SORT THREAD=REFERENCES MULTIAPPEND UNSELECT LITERAL+ CHILDREN NAMESPACE LOGIN-REFERRALS UIDPLUS LIST-EXTENDED I18NLEVEL=1 QUOTA imap_capability(pop3): imap_client_workarounds(default): outlook-idle delay-newmail imap_client_workarounds(imap): outlook-idle delay-newmail imap_client_workarounds(pop3): namespace: type: private prefix: INBOX. inbox: yes list: yes subscriptions: yes auth default: mechanisms: plain login digest-md5 user: vpopmail verbose: yes debug: yes passdb: driver: vpopmail userdb: driver: vpopmail plugin: quota: maildir
On Mar 21, 2009, at 7:22 AM, Jochen Schug wrote:
With the patch, I'm getting "User_over_quota._(#5.1.1)" errors. Mail
is delivered once and sender is sent a notification about the
exceeded quota. Without the patch, I get
"Temporary_error_on_maildir_delivery._(#4.3.0)". Mail is delivered,
but delivery is tried again and again. This only happens if a MUA is logged in to the mailbox and IMAP IDLE
is being used. If the MUA is disconnected: fine. If dovecot is
stopped: fine.
Is the problem that mail doesn't actually get deleted when IDLE is
used, or that a client being in IDLE is somehow causing qmail-local to
give the quota errors incorrectly? I can't see how the latter would be
possible, IDLE really has nothing to do with quota..
What IMAP client(s) are you using?
Is the problem that mail doesn't actually get deleted when IDLE is
used, or that a client being in IDLE is somehow causing qmail-local to
give the quota errors incorrectly? I can't see how the latter would be
possible, IDLE really has nothing to do with quota..
Just a thought...could it be that IDLE prevents a neccessary file handle from being released?
Sincerely,
- Wouter van der Schagt
On Mar 24, 2009, at 3:02 AM, Wouter van der Schagt wrote:
Is the problem that mail doesn't actually get deleted when IDLE is
used, or that a client being in IDLE is somehow causing qmail-local
to give the quota errors incorrectly? I can't see how the latter
would be possible, IDLE really has nothing to do with quota..Just a thought...could it be that IDLE prevents a neccessary file
handle from being released?
There is only one difference between what IDLE does and what a non-
IDLEing connection does: If inotify/dnotify/kqueue is enabled, Dovecot
notices immediately when a new mail pops up in new/ or cur/ directory
and does a resync. Hmm. Actually now I see one possibility if qmail-
local was written in a stupid/buggy way:
- qmail-local does rename tmp/file -> new/file
- Dovecot notices new/file and renames it to cur/file:2, immediately
- qmail-local stats new/file to see if rename() succeeded (not
necessary), notices it's no longer there and fails with quota failure
(..but why quota failure?..)
That's the only explanation I can think of. This could be tested by
compiling Dovecot with --with-notify=none configure option, which
would cause IDLE to check for new mails only once in a while and make
it much less likely to hit the race condition.
:2009-03-24T03:12:Timo Sirainen:
- qmail-local does rename tmp/file -> new/file
- Dovecot notices new/file and renames it to cur/file:2, immediately
- qmail-local stats new/file to see if rename() succeeded (not necessary), notices it's no longer there and fails with quota failure (..but why quota failure?..)
I'm guessing it's assuming that if the file doesn't exists it ran out of space(or has been limited by the quota) and thus returns the quota failure.
-- Andraž ruskie Levstik Source Mage GNU/Linux Games grimoire guru Geek/Hacker/Tinker
I may not agree with what you say, but I'll defend to the death your right to say it.
On Tue, 24 Mar 2009 08:31:34 +0100, Andraž 'ruskie' Levstik ruskie@codemages.net wrote:
:2009-03-24T03:12:Timo Sirainen:
- qmail-local does rename tmp/file -> new/file
- Dovecot notices new/file and renames it to cur/file:2, immediately
- qmail-local stats new/file to see if rename() succeeded (not
necessary), notices it's no longer there and fails with quota failure (..but why
quota failure?..)I'm guessing it's assuming that if the file doesn't exists it ran out of space(or has been limited by the quota) and thus returns the quota failure.
Could be something related to that. As I mentioned, an unpatched qmail-local also fails too, with a temporary delivery error and redelivers the mail again. So it seems to think that the mail was not delivered properly, but it shows up just fine in the IMAP folder.
Jochen
On Tue, 24 Mar 2009 08:12:07 +0100, Timo Sirainen tss@iki.fi wrote:
There is only one difference between what IDLE does and what a non- IDLEing connection does: If inotify/dnotify/kqueue is enabled, Dovecot
notices immediately when a new mail pops up in new/ or cur/ directory
and does a resync. Hmm. Actually now I see one possibility if qmail- local was written in a stupid/buggy way:
- qmail-local does rename tmp/file -> new/file
- Dovecot notices new/file and renames it to cur/file:2, immediately
- qmail-local stats new/file to see if rename() succeeded (not
necessary), notices it's no longer there and fails with quota failure
(..but why quota failure?..)
The quota failure comes from a patch. An unpatched qmail-local thinks the mail was not delivered properly, gives a temporary error and tries the delivery again... and again... and again.
I'm no C expert, therefore I can't tell how qmail-local handles the delivery exactly.
That's the only explanation I can think of. This could be tested by
compiling Dovecot with --with-notify=none configure option, which would
cause IDLE to check for new mails only once in a while and make it much
less likely to hit the race condition.
I tried that already, but that was at the very beginning of my troubleshooting. The only other reference to the problem I'm experiencing was from a guy using Debian, who got around it by installing the inotify packages.
Will --with-notify=none disable kevent and inotify/dnotify? I'll give it another shot then and report back.
Jochen
participants (4)
-
Andraž 'ruskie' Levstik
-
Jochen Schug
-
Timo Sirainen
-
Wouter van der Schagt