On Mon, 2007-03-12 at 19:53 +0000, John Lyons wrote:
I appreciate that being in a position where the indexes are corrupt is not good but then there may be unavoidable reasons why they become out of sync or corrupt.
Dovecot should fix all those problems internally. If it doesn't, it's a bug and if I can reproduce it I'll fix it. And they can't become "out of sync", because they're constantly synced with the backend.
If we add a new pop/imap server to the cluster is it to be expected that we'll have the service down for 5 or 6 hours while the new server creates indexes and chews through 100mbit of network capacity in the process?
I guess you use maildir? The dovecot-uidlist files are stored in NFS, right?
I think you have one or two problems which cause the disk I/O:
POP3 users who keep the mails stored in server. POP3 needs to get a list of all the mails' virtual sizes, which requires reading all the mails' contents. This could be avoided by adding ,W=<size> to the maildir filenames (http://wiki.dovecot.org/MailboxFormat/Maildir), although you can't do that for existing mails without causing their UIDs to change.
IMAP webmail, if you use one. They often use sorting/threading which requires reading all the mails' headers.
If this is all perfectly normal, I'd rather have a separate index building script that I can run on the file server and then export the built indexes to each of the pop servers to get things running again. I'm guessing indexes for 80Gb of mail could be built on a local file system in 30 to 60 mins rather than 6+ hours over a network?
Or just copy the indexes from another existing server?
I guess since you're using mostly POP3 the indexing script could be pretty simple (pop3 login/logout), but if you want it to do something useful for IMAP users you'll need to make it more complex (mailbox open/close doesn't update dovecot.index.cache file at all, which is really the only thing that matters much).