[Dovecot] maildir_file_do(/var/admhome/user/Maildir) racing
dovecot version 1.0.0, compile from the sources.
--------- snip -------- # /usr/local/etc/dovecot.conf listen: *:10143 ssl_listen: *:10943 ssl_cert_file: /etc/ssl/certs/postamt.pem ssl_key_file: /etc/ssl/private/postamt.key disable_plaintext_auth: no login_dir: /usr/local/var/run/dovecot/login login_executable: /usr/local/libexec/dovecot/imap-login mail_location: maildir:~/Maildir mail_plugins: quota imap_quota namespace: type: private separator: . prefix: INBOX. inbox: yes auth default: passdb: driver: pam userdb: driver: passwd plugin: quota: maildir quota: maildir:ignore=Trash --------- snip --------
We're using these odd ports, because there's still courier-imap on the original ports.
One, just one (!) of our users gets:
May 14 16:47:17 postamt dovecot: imap-login: Login: user=<username>, method=PLAIN, rip=141.42.x.y, lip=ip.of.the.server May 14 16:47:17 postamt dovecot: imap-login: Login: user=<username>, method=PLAIN, rip=141.42.x.y, lip=ip.of.the.server May 14 16:47:17 postamt dovecot: IMAP(username): maildir_file_do(/var/admhome/username/Maildir) racing May 14 16:47:17 postamt dovecot: IMAP(username): Disconnected: Internal error occurred. Refer to server log for more information. [2007-05-14 16:47:17]
Now it's a bit hard for me to see what's going on. /var/admhome/username/Maildir is on /, thus:
$ mount /dev/cciss/c0d0p6 on / type ext3 (rw,errors=panic)
$ dmesg does not complain about any corruption
$ uname -a Linux postamt.charite.de 2.6.21.1 #1 SMP Tue May 1 15:50:11 CEST 2007 i686 GNU/Linux
OS: Debian testing
-- Ralf Hildebrandt (i.A. des IT-Zentrums) Ralf.Hildebrandt@charite.de Charite - Universitätsmedizin Berlin Tel. +49 (0)30-450 570-155 Gemeinsame Einrichtung von FU- und HU-Berlin Fax. +49 (0)30-450 570-962 IT-Zentrum Standort CBF send no mail to plonk@charite.de
On 14.5.2007, at 17.54, Ralf Hildebrandt wrote:
One, just one (!) of our users gets:
May 14 16:47:17 postamt dovecot: imap-login: Login:
user=<username>, method=PLAIN, rip=141.42.x.y, lip=ip.of.the.server May 14 16:47:17 postamt dovecot: imap-login: Login:
user=<username>, method=PLAIN, rip=141.42.x.y, lip=ip.of.the.server May 14 16:47:17 postamt dovecot: IMAP(username): maildir_file_do(/ var/admhome/username/Maildir) racing
This happens at every login? I thought I'd never see this racing
error. It means that Dovecot tried to some operation to a maildir
file. But the operation failed, because the file was just renamed
(eg. flag changed). So it located the file again, and found it. Then
it tried the operation again, but it failed the same way. This was
tried 10 times until it gave this "racing" error.
I think it's a bit unlikely that the filename was changing that fast.
So there's probably a bug somewhere.. Could get strace output from
those two logins and send them to me? You can do this with eg.:
mail_executable = /usr/local/bin/run-imap.sh
run-imap.sh:
#!/bin/sh
exec strace -tt -s 1000 -o /tmp/imap.$USER.time +%u.$$.log /usr/
local/libexec/dovecot/imap
- Timo Sirainen <tss@iki.fi>:
This happens at every login? I thought I'd never see this racing
error.
I found the reason:
The User had a symlink in his $HOME/Maildir/cur, which pointed to a NON-existing file :(
Courier-IMAP ignored that, dovecot bailed out.
I removed the bogus symlink and all is well now.
It means that Dovecot tried to some operation to a maildir
file.
Yep.
-- Ralf Hildebrandt (i.A. des IT-Zentrums) Ralf.Hildebrandt@charite.de Charite - Universitätsmedizin Berlin Tel. +49 (0)30-450 570-155 Gemeinsame Einrichtung von FU- und HU-Berlin Fax. +49 (0)30-450 570-962 IT-Zentrum Standort CBF send no mail to plonk@charite.de
On Tue, 2007-05-15 at 11:48 +0200, Ralf Hildebrandt wrote:
- Timo Sirainen <tss@iki.fi>:
This happens at every login? I thought I'd never see this racing
error.I found the reason:
The User had a symlink in his $HOME/Maildir/cur, which pointed to a NON-existing file :(
I've changed Dovecot's behavior now also:
x fetch * body imap(tss): Error: Maildir: Symlink destination doesn't exist: /home/tss/Maildir/cur/dgdgd
- 40 FETCH (BODY ("text" "plain" NIL NIL NIL "7bit" 0 0 NIL NIL NIL)) x OK Fetch completed.
So it logs an error but still goes ahead without disconnecting the client.
- Timo Sirainen <tss@iki.fi>:
I've changed Dovecot's behavior now also:
x fetch * body imap(tss): Error: Maildir: Symlink destination doesn't exist: /home/tss/Maildir/cur/dgdgd
- 40 FETCH (BODY ("text" "plain" NIL NIL NIL "7bit" 0 0 NIL NIL NIL)) x OK Fetch completed.
So it logs an error but still goes ahead without disconnecting the client.
Great! Thank you.
Success story: We were using courier-imap with 9000 User accounts here at Charite and yesterday I switched to dovecot for imap, imaps and pop3s. The load dropped from 50-60 to 2 (!) during noon.
Excellent!
-- Ralf Hildebrandt (i.A. des IT-Zentrums) Ralf.Hildebrandt@charite.de Charite - Universitätsmedizin Berlin Tel. +49 (0)30-450 570-155 Gemeinsame Einrichtung von FU- und HU-Berlin Fax. +49 (0)30-450 570-962 IT-Zentrum Standort CBF send no mail to plonk@charite.de
Hello Ralf,
Ralf Hildebrandt <Ralf.Hildebrandt@charite.de>, [20070515 - 12:37:36]
- Timo Sirainen <tss@iki.fi>:
I've changed Dovecot's behavior now also:
x fetch * body imap(tss): Error: Maildir: Symlink destination doesn't exist: /home/tss/Maildir/cur/dgdgd
- 40 FETCH (BODY ("text" "plain" NIL NIL NIL "7bit" 0 0 NIL NIL NIL)) x OK Fetch completed.
So it logs an error but still goes ahead without disconnecting the client.
Great! Thank you.
Success story: We were using courier-imap with 9000 User accounts here at Charite and yesterday I switched to dovecot for imap, imaps and pop3s. The load dropped from 50-60 to 2 (!) during noon.
Excellent!
Are you willing to elaborate a bit about your setup, hardware, config, etc? This would be good to know what mailsystem is behind that values :-)
Thanks Stephan
- Stephan Holl <stephan@holl-land.de>:
Are you willing to elaborate a bit about your setup, hardware, config, etc? This would be good to know what mailsystem is behind that values :-)
Debian/testing, hand-rolled 2.6.21.1-NOHZ-SMP kernel The users are stored in /etc/passwd (for historical reasons (don't ask)) Dual Xeon 2.8GHz with 3GB RAM 1,8TB mailstore (ext3, external SCSI RAID)
-- Ralf Hildebrandt (i.A. des IT-Zentrums) Ralf.Hildebrandt@charite.de Charite - Universitätsmedizin Berlin Tel. +49 (0)30-450 570-155 Gemeinsame Einrichtung von FU- und HU-Berlin Fax. +49 (0)30-450 570-962 IT-Zentrum Standort CBF send no mail to plonk@charite.de
participants (3)
-
Ralf Hildebrandt
-
Stephan Holl
-
Timo Sirainen