[Dovecot] index performance issues

[Timo Sirainen]

On Mon, 2007-03-12 at 19:53 +0000, John Lyons wrote:

> I appreciate that being in a position where the indexes are corrupt is
> not
> good but then there may be unavoidable reasons why they become out of
> sync
> or corrupt.

Dovecot should fix all those problems internally. If it doesn't, it's a
bug and if I can reproduce it I'll fix it. And they can't become "out of
sync", because they're constantly synced with the backend.

> If we add a new pop/imap server to the cluster is it to be expected
> that
> we'll have the service down for 5 or 6 hours while the new server
> creates
> indexes and chews through 100mbit of network capacity in the process?

I guess you use maildir? The dovecot-uidlist files are stored in NFS,
right?

I think you have one or two problems which cause the disk I/O:

1. POP3 users who keep the mails stored in server. POP3 needs to get a
list of all the mails' virtual sizes, which requires reading all the
mails' contents. This could be avoided by adding ,W=<size> to the
maildir filenames (http://wiki.dovecot.org/MailboxFormat/Maildir),
although you can't do that for existing mails without causing their UIDs
to change.

2. IMAP webmail, if you use one. They often use sorting/threading which
requires reading all the mails' headers.

> If this is all perfectly normal, I'd rather have a separate index
> building
> script that I can run on the file server and then export the built
> indexes
> to each of the pop servers to get things running again.  I'm guessing
> indexes for 80Gb of mail could be built on a local file system in 30
> to 60
> mins rather than 6+ hours over a network? 

Or just copy the indexes from another existing server?

I guess since you're using mostly POP3 the indexing script could be
pretty simple (pop3 login/logout), but if you want it to do something
useful for IMAP users you'll need to make it more complex (mailbox
open/close doesn't update dovecot.index.cache file at all, which is
really the only thing that matters much).
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url : http://dovecot.org/pipermail/dovecot/attachments/20070312/f861e21f/attachment.pgp