[Dovecot] Dovecot index, NFS, and multiple architectures

[David Halik]

I just reproduced the environment and the index corrupted immediately 
across NFS because of the endian issue.

Jun 25 11:53:34 host IMAP(user): : Rebuilding index file 
/dovecot-index/index/user/.INBOX/dovecot.index: CPU architecture changed
Jun 25 11:53:35 host IMAP(user): : Corrupted index cache file 
/dovecot-index/index/user/.INBOX/dovecot.index.cache: field header 
points outside file

This was starting from a clean index, first opening pine on the NFS 
Solaris 9 sparc machine, and then at the same time opening pine on my 
Fedora 9 i386 workstation.

I'm going to try the idea of splitting the indexes into two different 
architectures, but I'm worried that this will not be feasible when we 
try to scale to our 80,000 users.

By the way, I don't think this is related to the corruption, but we also 
have tons of these in the logs:

Jun 25 11:52:32 host IMAP(user): : Created dotlock file's timestamp is 
different than current time (1214409234 vs 1214409152): 
/dovecot-index/control/user/.INBOX/dovecot-uidlist
Jun 25 11:52:32 host IMAP(user): : Created dotlock file's timestamp is 
different than current time (1214409235 vs 1214409152): 
/dovecot-index/control/user/.INBOX/dovecot-uidlist


Timo Sirainen wrote:
> On Jun 18, 2008, at 9:12 PM, David Halik wrote:
>
>> Now this setup is just a test example and not exactly what we'll be 
>> running in production, but it tipped up the problem either way. Since 
>> the index is shared by both the Linux i386 machine and the sparc64 
>> Solaris machine, if mail is accessed from both, lets say with Pine 
>> for example, the index becomes corrupted and breaks. As long as only 
>> one architecture only ever touches it there are no issues.
>
> It should only log "architecture changed" and rebuild the index. I did 
> some fixed related to this at some point though, might have been only 
> for v1.1.
>
>> I'm assuming this is an endian issue, which would make the most 
>> sense. Is there a way around this with flags or server options? Is 
>> this something that has maybe been addressed in 1.1.0?
>
> No. I once thought about making it endianess-independent, but rejected 
> the idea soon after I figured out that it would unnecessarily slow 
> down nearly-100% of users and the code would become really ugly.
>
> Although this reminds me .. dbox does rely on indexes more than 
> mbox/maildir. Perhaps there should be code that attempts to preserve 
> and convert the important data (flags, keywords) from different 
> endianess indexes. But this still wouldn't make it possible to quickly 
> switch between architectures.
>
> One thing you could do is to specify different index files paths for 
> your installations. For example for sparc64:
>
> mail_location = maildir:~/Maildir:INDEX=~/Maildir/sparc64
>


-- 
================================
David Halik
System Administrator
OIT-CSS Rutgers University
dhalik at jla.rutgers.edu
================================