[Dovecot] All index file bugs fixed(?)
I fixed the nightly snapshot building. Now it's again being built every night (if there have been any changes). Please test it before I'll make rc18 release. http://dovecot.org/nightly/dovecot-latest.tar.gz
I've also been stress testing a lot with http://dovecot.org/tools/imaptest.c. Last night I ran it for 9 hours with mmap_disable=yes and it picked up only one race condition, which is also now fixed. I'll run the next night with mmap_disable=no and see if anything shows up with it.
I'm hopeful that there aren't any important index file related bugs left anymore. This is actually the first time in Dovecot's existence that I'm really not aware of *any* bugs in index file handling.
v1.0 release should be really near now, unless you figure out new problems :)
Here's a list of bugs that should have been fixed since rc17:
"Corrupted index file dovecot.index: Record field keywords points outside record size (12 < 12+2)" "Corrupted Index file dovecot.index: Record [1].uid=0" Crashes
"open() failed with file dovecot.index.log: No such file or directory"
"Fixed index file dovecot.index: seen_messages_count 8 -> 9" "View synchronization from transaction log failed: Deleted counter wrong" etc.
Corrupted transaction log errors:
- http://dovecot.org/list/dovecot-cvs/2007-January/007389.html
- http://dovecot.org/list/dovecot-cvs/2007-January/007393.html
Corrupted view errors:
- http://dovecot.org/list/dovecot-cvs/2007-January/007366.html
- http://dovecot.org/list/dovecot-cvs/2007-January/007368.html
Corrupted cache file errors:
Fixed a lot of crashes whenever something went wrong (I deleted index files every 3 seconds until it crashed, last run went for several minutes before I stopped trying):
Hi Timo, im not seeing anymore crashes in the last 12 hours or so. Whenever you sent me that patch. So this looks very good!
Cor
On Wed, 2007-01-17 at 21:33 +0200, Timo Sirainen wrote:
Here's a list of bugs that should have been fixed since rc17:
Found one more:
Keywords were sometimes being re-added to the index even while they already existed, causing the record size to bloat.
New snapshot is ready in a few minutes.
Timo Sirainen wrote:
I fixed the nightly snapshot building. Now it's again being built every night (if there have been any changes). Please test it before I'll make rc18 release. http://dovecot.org/nightly/dovecot-latest.tar.gz
I've also been stress testing a lot with http://dovecot.org/tools/imaptest.c. Last night I ran it for 9 hours with mmap_disable=yes and it picked up only one race condition, which is also now fixed. I'll run the next night with mmap_disable=no and see if anything shows up with it.
I'm hopeful that there aren't any important index file related bugs left anymore. This is actually the first time in Dovecot's existence that I'm really not aware of *any* bugs in index file handling.
v1.0 release should be really near now, unless you figure out new problems :)
Here's a list of bugs that should have been fixed since rc17:
"Corrupted index file dovecot.index: Record field keywords points outside record size (12 < 12+2)" "Corrupted Index file dovecot.index: Record [1].uid=0" Crashes
"open() failed with file dovecot.index.log: No such file or directory"
"Fixed index file dovecot.index: seen_messages_count 8 -> 9" "View synchronization from transaction log failed: Deleted counter wrong" etc.
Corrupted transaction log errors:
- http://dovecot.org/list/dovecot-cvs/2007-January/007389.html
- http://dovecot.org/list/dovecot-cvs/2007-January/007393.html
Corrupted view errors:
- http://dovecot.org/list/dovecot-cvs/2007-January/007366.html
- http://dovecot.org/list/dovecot-cvs/2007-January/007368.html
Corrupted cache file errors:
Fixed a lot of crashes whenever something went wrong (I deleted index files every 3 seconds until it crashed, last run went for several minutes before I stopped trying):
it happened again (with nightly/dovecot-latest):
Error: IMAP(rainer): Fixed index file /home/rainer/Maildir/.INBOX.amavisd/dovecot.index: first_recent_uid_lowwater 199 -> 198
-- in diesem Sinne, Rainer
On Fri, 2007-01-19 at 23:55 +0100, Rainer Ophoven wrote:
Error: IMAP(rainer): Fixed index file /home/rainer/Maildir/.INBOX.amavisd/dovecot.index: first_recent_uid_lowwater 199 -> 198
Wonder how this problem could be reproduced.. How often does it happen to you? Does the error happen when you're doing something actively for the mailbox, or only when you're away idling? What happens to the INBOX.amavisd mailbox in general? How do the mails get there (Dovecot deliver, IMAP copy, something else)? Do you delete mails from it?
Have you tried if mmap_disable=yes removes this problem?
Timo Sirainen wrote:
On Fri, 2007-01-19 at 23:55 +0100, Rainer Ophoven wrote:
Error: IMAP(rainer): Fixed index file /home/rainer/Maildir/.INBOX.amavisd/dovecot.index: first_recent_uid_lowwater 199 -> 198
Wonder how this problem could be reproduced.. How often does it happen to you? Does the error happen when you're doing something actively for the mailbox, or only when you're away idling? What happens to the INBOX.amavisd mailbox in general? How do the mails get there (Dovecot deliver, IMAP copy, something else)? Do you delete mails from it?
Have you tried if mmap_disable=yes removes this problem?
It happens every 2nd day or so , sometimes two or three times a day, then it won't happen for a few days at all.
Tonight it happened, after restarting my windows-box from hibernation, but I closed Thunderbird explicitely before XP went into hibernation.
the messages get to INBOX.amavisd and INBOX.dovecot by rules that I set up in Thunderbird depending on "subject" and "from". It's meant as a means to sort messages on arrival (more or less). the mails are originally delivered to INBOX by postfix - not using dovecot-deliver.
I sure delete mails from these mailboxes, maybe sometimes while Thunderbird ist still in the process of refreshing and moving mails from INBOX to the two sub-mailboxes.
I have three
-- in diesem Sinne, Rainer
On Sat, 2007-01-20 at 00:25 +0100, Rainer Ophoven wrote:
Timo Sirainen wrote:
On Fri, 2007-01-19 at 23:55 +0100, Rainer Ophoven wrote:
Error: IMAP(rainer): Fixed index file /home/rainer/Maildir/.INBOX.amavisd/dovecot.index: first_recent_uid_lowwater 199 -> 198
Wonder how this problem could be reproduced.. How often does it happen to you? Does the error happen when you're doing something actively for the mailbox, or only when you're away idling? What happens to the INBOX.amavisd mailbox in general? How do the mails get there (Dovecot deliver, IMAP copy, something else)? Do you delete mails from it?
Have you tried if mmap_disable=yes removes this problem?
It happens every 2nd day or so , sometimes two or three times a day, then it won't happen for a few days at all.
Tonight it happened, after restarting my windows-box from hibernation, but I closed Thunderbird explicitely before XP went into hibernation.
the messages get to INBOX.amavisd and INBOX.dovecot by rules that I set up in Thunderbird depending on "subject" and "from". It's meant as a means to sort messages on arrival (more or less). the mails are originally delivered to INBOX by postfix - not using dovecot-deliver.
I sure delete mails from these mailboxes, maybe sometimes while Thunderbird ist still in the process of refreshing and moving mails from INBOX to the two sub-mailboxes.
I have three
Hi.
Your setup looks just like mine here, so i just checked my logs and i can see that i have the same in the logs as you have.
I have fetchmail -> postfix -> maildir
and i have the mail client do the sorting too, which also has been thunderbird for a period, and is now evolution again.
niki2 dovecot # grep "Fixed" * log-2007-01-02-01:00:40:Jan 1 22:30:02 [dovecot] IMAP(niki): Fixed index file /home/niki/.maildir_aux/.INBOX.Mail Lists.Gentoo.Dev/dovecot.index: first_recent_uid_lowwater 400 -> 399 log-2007-01-02-01:00:40:Jan 2 00:51:13 [dovecot] IMAP(niki): Fixed index file /home/niki/.maildir_aux/.INBOX.Mail Lists.Gentoo.Hardened/dovecot.index: first_recent_uid_lowwater 31 -> 30 log-2007-01-03-02:36:02:Jan 2 01:00:40 [dovecot] IMAP(niki): Fixed index file /home/niki/.maildir_aux/.INBOX.Mail Lists.Gentoo.Dev/dovecot.index: first_recent_uid_lowwater 411 -> 409 log-2007-01-03-02:36:02:Jan 2 04:00:42 [dovecot] IMAP(niki): Fixed index file /home/niki/.maildir_aux/.INBOX.Mail Lists.Gentoo.Dev/dovecot.index: first_recent_uid_lowwater 422 -> 420 log-2007-01-03-02:36:02:Jan 2 23:22:55 [dovecot] IMAP(niki): Fixed index file /home/niki/.maildir_aux/.INBOX.Mail Lists.Gentoo.Dev/dovecot.index: first_recent_uid_lowwater 430 -> 428 log-2007-01-06-08:59:50:Jan 5 16:00:31 [dovecot] IMAP(niki): Fixed index file /home/niki/.maildir_aux/.INBOX.Mail Lists.Dovecot.cvs/dovecot.index: first_recent_uid_lowwater 1044 -> 1043
And this is the installation dates of dovecot, so it spans 3 versions, don't have logs further back, so don't know if any older versions are affected.
Wed Dec 27 13:16:16 2006 >>> net-mail/dovecot-1.0_rc15-r1
Sat Jan 6 09:08:37 2007 >>> net-mail/dovecot-1.0_rc16
Tue Jan 9 01:35:17 2007 >>> net-mail/dovecot-1.0_rc17
I have "mail_read_mmaped = yes" and the rest of the mmap stuff is commented out.
the environment is linux kernel 2.6.19-gentoo-r4, on a x86_64 laptop.
And i haven't tried the snapshot yet.
Regards
Niki
On Sat, 2007-01-20 at 15:58 +0100, Niki Guldbrand wrote:
log-2007-01-02-01:00:40:Jan 1 22:30:02 [dovecot] IMAP(niki): Fixed index file /home/niki/.maildir_aux/.INBOX.Mail Lists.Gentoo.Dev/dovecot.index: first_recent_uid_lowwater 400 -> 399
Niki, Rainer, do you also have delay-newmail in workarounds?
Timo Sirainen wrote:
On Sat, 2007-01-20 at 15:58 +0100, Niki Guldbrand wrote:
log-2007-01-02-01:00:40:Jan 1 22:30:02 [dovecot] IMAP(niki): Fixed index file /home/niki/.maildir_aux/.INBOX.Mail Lists.Gentoo.Dev/dovecot.index: first_recent_uid_lowwater 400 -> 399
Niki, Rainer, do you also have delay-newmail in workarounds?
no I don't - just outlook-idle
-- in diesem Sinne, Rainer
On Tue, 2007-01-23 at 18:37 +0200, Timo Sirainen wrote:
On Sat, 2007-01-20 at 15:58 +0100, Niki Guldbrand wrote:
log-2007-01-02-01:00:40:Jan 1 22:30:02 [dovecot] IMAP(niki): Fixed index file /home/niki/.maildir_aux/.INBOX.Mail Lists.Gentoo.Dev/dovecot.index: first_recent_uid_lowwater 400 -> 399
Niki, Rainer, do you also have delay-newmail in workarounds?
No, i don't have any workarounds enabled.
Niki
Rainer Ophoven wrote:
Timo Sirainen wrote:
On Fri, 2007-01-19 at 23:55 +0100, Rainer Ophoven wrote:
Error: IMAP(rainer): Fixed index file /home/rainer/Maildir/.INBOX.amavisd/dovecot.index: first_recent_uid_lowwater 199 -> 198 Wonder how this problem could be reproduced.. How often does it happen to you? Does the error happen when you're doing something actively for the mailbox, or only when you're away idling? What happens to the INBOX.amavisd mailbox in general? How do the mails get there (Dovecot deliver, IMAP copy, something else)? Do you delete mails from it?
Have you tried if mmap_disable=yes removes this problem?
It happens every 2nd day or so , sometimes two or three times a day, then it won't happen for a few days at all.
Tonight it happened, after restarting my windows-box from hibernation, but I closed Thunderbird explicitely before XP went into hibernation.
the messages get to INBOX.amavisd and INBOX.dovecot by rules that I set up in Thunderbird depending on "subject" and "from". It's meant as a means to sort messages on arrival (more or less). the mails are originally delivered to INBOX by postfix - not using dovecot-deliver.
I sure delete mails from these mailboxes, maybe sometimes while Thunderbird ist still in the process of refreshing and moving mails from INBOX to the two sub-mailboxes.
I have three
I always thought it'd only happened with Thunderbird but tonight it happened with Mulberry-Client as well. Error: IMAP(rainer): Fixed index file /home/rainer/Maildir/.INBOX.thunderbird/dovecot.index: first_recent_uid_lowwater 23 -> 22
in diesem Sinne, Rainer
On 1/20/07, Rainer Ophoven <rainer@route66.homelinux.net> wrote: ...
I always thought it'd only happened with Thunderbird but tonight it
happened with Mulberry-Client as well. Error: IMAP(rainer): Fixed index file /home/rainer/Maildir/.INBOX.thunderbird/dovecot.index: first_recent_uid_lowwater 23 -> 22
Last Tuesday(16th) I enabled maildir_copy_with_hardlinks, and now I get these messages as well(never saw them with maildir_copy_with_hardlinks=no). It has happened 130 times since the 16th according to the logs.
RHEL4.3 NFS, mmap_disable=yes, lock_method=fcntl, indices/maildir on netapp.
dovecot: Jan 16 12:06:49 Error: IMAP(user1): Fixed index file mailstore/user1/Maildir/.Trash/dovecot.index: first_recent_uid_lowwater 2977 -> 2971 dovecot: Jan 16 13:57:22 Error: IMAP(user2): Fixed index file mailstore/user2/Maildir/.Trash/dovecot.index: first_recent_uid_lowwater 1920 -> 1917 dovecot: Jan 18 14:13:29 Error: IMAP(user3): Fixed index file mailstore/user3/Maildir/.svn/dovecot.index: first_recent_uid_lowwater 1601 -> 1600
On 1/23/07, Timo Sirainen <tss@iki.fi> wrote:
On Mon, 2007-01-22 at 18:44 -0500, bofh list wrote:
dovecot: Jan 16 12:06:49 Error: IMAP(user1): Fixed index file mailstore/user1/Maildir/.Trash/dovecot.index: first_recent_uid_lowwater 2977 -> 2971
Could you try with the attached patch and tell me what it logs?
I will. It will take some time to get into production though.
Thanks again for your help!
On 1/23/07, Timo Sirainen <tss@iki.fi> wrote:
On Mon, 2007-01-22 at 18:44 -0500, bofh list wrote:
dovecot: Jan 16 12:06:49 Error: IMAP(user1): Fixed index file mailstore/user1/Maildir/.Trash/dovecot.index: first_recent_uid_lowwater 2977 -> 2971
Do you use delay-newmail workaround?
Yes, here are the workaround we use:
imap_client_workarounds = outlook-idle delay-newmail
participants (5)
-
bofh list
-
Cor Bosman
-
Niki Guldbrand
-
Rainer Ophoven
-
Timo Sirainen