[dovecot] Maildir syncing rewrite
It finally seems to be working. I've done some testing now and it seems to be working fine. If you have time, see if you can get it to break (especially with INDEX=memory or by manually modifying the maildir).
The UIDs are kept in Courier-compatible dovecot-uidlist file (so "mv courierimapuiddb dovecot-uidlist" should work). The great thing about this code is that we never wait for lock. If we can't lock, we simply use the old uidlist file to sync whatever we can, which should be everything but the very newest mails just being delivered and processed by another Dovecot. Of course, if they share index files, we're still blocking there.
I named the file as "dovecot-uidlist" instead of ".uidlist" because I finally realized that files beginning with dot conflict with the mailbox namespace. I should rename .customflags and .subscriptions files as well. Also if I move .imap.index* files into root dir, I'll have to rename them too.
new/ directory is now scanned every time to see if there's new mail. cur/ directory is scanned if index file's timestamp differs from cur/ dir's timestamp. uidlist file is reread if it's device/inode changes.
You can get the sources from either CVS, or 0.99.9-test2 from http://dovecot.procontrol.fi/test/
At 10 Apr 2003 00:03:14 +0300, Timo Sirainen tss@iki.fi wrote:
You can get the sources from either CVS, or 0.99.9-test2 from http://dovecot.procontrol.fi/test/
Wow, it's much faster! ...but sometimes it dies with the following syslog message. (locking broken?)
Apr 10 15:58:53 hadaly imap(fuyuki): Corrupted index file /home/fuyuki/Maildir/.INBOX/.imap.index: index.next_uid (12797) > uidlist. next_uid (12790) Apr 10 15:58:53 hadaly dovecot: child 71956 (imap) killed with signal 11
I'm using flock for both dovecot and postfix, which is my local mail delivery agent.
-- fuyuki
On Thu, 2003-04-10 at 10:14, Kimura Fuyuki wrote:
You can get the sources from either CVS, or 0.99.9-test2 from http://dovecot.procontrol.fi/test/
Wow, it's much faster! ...but sometimes it dies with the following syslog message. (locking broken?)
Faster? How large mailboxes or slow computer do you have? :) I don't think I made it any faster, and I don't think it was slow before either.. It took more memory though.
Apr 10 15:58:53 hadaly imap(fuyuki): Corrupted index file /home/fuyuki/Maildir/.INBOX/.imap.index: index.next_uid (12797) > uidlist. next_uid (12790) Apr 10 15:58:53 hadaly dovecot: child 71956 (imap) killed with signal 11
I'm using flock for both dovecot and postfix, which is my local mail delivery agent.
flock? With maildir? There's no locks with maildir, except for uidlist.lock file which relies on O_EXCL working for open(). Indexes and others are locked with fcntl(), unless you've directly changed that from sources.
But I'm not really sure why the above happens.. Even the crash shouldn't happen if the corruption happens (I forced corruption, worked fine). It would help if you could get gdb backtrace, core dumps can be enabled by setting "mail_drop_priv_before_exec = yes" in config file.
On 10 Apr 2003, Timo Sirainen wrote:
You can get the sources from either CVS, or 0.99.9-test2 from http://dovecot.procontrol.fi/test/ I'm using flock for both dovecot and postfix, which is my local mail delivery agent. flock? With maildir? There's no locks with maildir, except for uidlist.lock file which relies on O_EXCL working for open(). Indexes and others are locked with fcntl(), unless you've directly changed that from
On Thu, 2003-04-10 at 10:14, Kimura Fuyuki wrote: sources.
For the record, is out-of-the-box Dovecot+Maildir with indexes NFS-safe?
Andy
-- Andreas Aardal Hanssen | http://www.andreas.hanssen.name/gpg Author of Binc IMAP | Nil desperandum
On Fri, 2003-04-11 at 00:01, Andreas Aardal Hanssen wrote:
For the record, is out-of-the-box Dovecot+Maildir with indexes NFS-safe?
Index files will probably never be NFS-safe. They can be built into memory which should be quite close to how other IMAP servers work, but I don't think I'll do that by default.
Other than that, next release should be NFS-safe out of the box.
At 10 Apr 2003 23:44:23 +0300, Timo Sirainen tss@iki.fi wrote:
Faster? How large mailboxes or slow computer do you have? :) I don't think I made it any faster, and I don't think it was slow before either.. It took more memory though.
Yes, I'm surprised too. I have no idea how it could be done, but it's really faster. (why??)
This is my slow computer. :)
OS: FreeBSD 5.0 CPU: K6-2 400MHz MUA: Wanderlust 2.10.0
And some mailboxes have 1000+ mails.
But I'm not really sure why the above happens.. Even the crash shouldn't happen if the corruption happens (I forced corruption, worked fine). It would help if you could get gdb backtrace, core dumps can be enabled by setting "mail_drop_priv_before_exec = yes" in config file.
OK, I'll try it if I have time, but I suspect the mount option -o noatime would be the source of the problem... (I've dropped the option.)
BTW, Here's another ominous message.
Apr 11 08:12:08 hadaly imap(fuyuki): Corrupted index file (in-memory index for /home/fuyuki/Maildir): Filename mismatch for UID 1: 1050013385.V40dI4d173.hadaly.dyndns.org vs 1049984584.V40dI4d1f8.hadaly.dyndns.org
-- fuyuki
participants (3)
-
Andreas Aardal Hanssen
-
Kimura Fuyuki
-
Timo Sirainen