[Dovecot] bidirectional synchronization of dovecot-managed maildirs
Hello and happy holidays,
I've been thinking about high performance bidirectional synchronizing the mail on my laptop (which is frequently disconnected) and my workstation at work (which is always on and receives all inbound mail). Both run dovecot on top of maildir.
I've been playing with offlineimap, unison (which I already use for all my non-mail files), as well as the gnus agent (in IMAP "disconnected mode"). None of these solutions is really satisfactory:
offlineimap treats a message refile between folders as a delete-plus-resend;
unison doesn't understand the semantics of flag changes at the end of files in maildirs; it also has to stat all the files (which can become slow) to detect changes;
gnus agent can't refile messages between folders when in disconnected mode.
Perhaps there are better disconnected IMAP modes available for other MUAs, but, it seems like a more general solution would be appropriate.
In particular, the "uniq" field of a maildir filename is an easy way of tracking the identity of a message as it gets refiled. My question: would it be difficult to patch dovecot so that it maintained the *same* uniq when copying messages between folders?
What about adding hooks that get triggered whenever dovecot copies or deletes messages? These hooks could log the changes. Later, when one tries to sync, one merges the changes from both replicas and then applies them.
I'm aware that dovecot has an option to do copies as hardlinks, which is a good start, but still requires one to stat the files to discover which have the same inode. This is *much* slower than simply listing the directories and comparing maildir "uniq" fields.
I'm aware that this invariant wouldn't be maintained if other programs modified the maildirs under dovecot's nose, but I'm willing to exclude this possibility.
Suggestions?
Best regards, -James Leifer
Marcus Rueckert rueckert@informatik.uni-rostock.de writes:
kmail - offline imap mailsync
Hi Marcus,
Thanks for your message. Looked at both your suggestions, neither of which really do what I want. 8-)
kmail: I don't really want to be tied down to a particular MUA
mailsync: seems to still require opening messages to find their message-ids --- something that's much slower than just using the "uniq" part of the filename to track the identity of messages as they are moved. Also I don't fully trust message-ids in these days of spam since spammers can reuse any old message-ids they find.
So my question still stands! Would it be difficult to hack dovecot so that it maintained the "uniq" part of a maildir file name when it copied it?
While I'm asking this: how difficult would it be to have multiple physical maildir directories be presented by dovecot as a single IMAP folder? The reason I ask is that many file systems start to slow down noticably once one gets more than about 5000 files in a directory. (Reiser is an exception, but reiser has had other problems of reliability for me.) It would be nice to be able to spread a single folder out over several maildirs to reduce any possible problems from the file system.
Warm regards, --James Leifer
On Tue, 2004-12-28 at 12:13 +0100, James Leifer wrote:
While I'm asking this: how difficult would it be to have multiple physical maildir directories be presented by dovecot as a single IMAP folder? The reason I ask is that many file systems start to slow down noticably once one gets more than about 5000 files in a directory. (Reiser is an exception, but reiser has had other problems of reliability for me.) It would be nice to be able to spread a single folder out over several maildirs to reduce any possible problems from the file system.
Don't pretty much all Linux filesystems handle this well already? Even ext3 is fast nowadays with htree enabled.
This isn't really maildir anymore, maybe rather have a completely new mail store format. I'll probably build one somewhat soon, featuring:
- keep message flags, UIDs and other metadata in Dovecot indexes
- 1..n messages in a single file, eg. 1MB limit per file (unless a single message itself is larger)
- expunging in the middle of file would move the following messages to new file, so it can't get corrupted
- no locks needed for reading
On Mon, 2004-12-27 at 16:11 +0100, James Leifer wrote:
- offlineimap treats a message refile between folders as a delete-plus-resend;
What other way is there to treat it? IMAP doesn't track message copies across folders in any case.
In particular, the "uniq" field of a maildir filename is an easy way of tracking the identity of a message as it gets refiled. My question: would it be difficult to patch dovecot so that it maintained the *same* uniq when copying messages between folders?
I don't think it's a good idea. Then copying a message into folder, setting a flag for it, and copying the same message there again would create maildir file with duplicate unique part.
participants (3)
-
James Leifer
-
Marcus Rueckert
-
Timo Sirainen