[Dovecot] Dovecot index, NFS, and multiple architectures
Hi all,
I crawled through the archives for a bit but didn't see anything helpful, so I apologize if this has already been addressed. We've been dying to move from Courier to Dovecot across our whole infrastructure for quite some time, but until recently our setup wasn't possible until this happened:
"Dovecot allows mailboxes and their indexes to be modified by multiple computers at the same time, while still performing well. This means that Dovecot works with NFS and clustered filesystems."
Now that NFS is officially supported, we figured why not make the switch. All of our mail touches NFS in some way, so we need to check on the stability before completely migrating. In test trials we're having issues with the NFS'd index becoming corrupted. Here's the setup:
- Linux workstations running Fedora 8/9 i386 and a locally called Dovecot 1.0.14
- NFS'd homedir with Maildir setup
- NFS is on Solaris 9 sparcv9 (64bit) running Dovecot 1.0.14
Now this setup is just a test example and not exactly what we'll be running in production, but it tipped up the problem either way. Since the index is shared by both the Linux i386 machine and the sparc64 Solaris machine, if mail is accessed from both, lets say with Pine for example, the index becomes corrupted and breaks. As long as only one architecture only ever touches it there are no issues.
I'm assuming this is an endian issue, which would make the most sense. Is there a way around this with flags or server options? Is this something that has maybe been addressed in 1.1.0?
Unfortunately, we are not going to be able to move to Dovecot across all of our systems until this is no longer an issue. We run a lot of mixed environments that have everything from Linux i386/x86_64 to Solaris 9 64 to Windows. If a user accesses IMAP from a Windows box, then logs into the front end which is a Linux x86_64 box and runs Pine, and all of this is on a Solaris sparc NFS system, we're going to have serious issues with the index. ;)
Any help would be appreciated. Thanks, -Dave
--
David Halik System Administrator OIT-CSS Rutgers University dhalik@jla.rutgers.edu http://www.digitalruin.net
On 6/18/2008, David Halik (dhalik@jla.rutgers.edu) wrote:
- Linux workstations running Fedora 8/9 i386 and a locally called Dovecot 1.0.14
- NFS'd homedir with Maildir setup
- NFS is on Solaris 9 sparcv9 (64bit) running Dovecot 1.0.14
NFS is only fully supported on 1.1+. This is why you're having trouble.
1.1rc10 is the latest, and the stable release is imminent, so you should have no trouble moving to it.
--
Best regards,
Charles
Great, I was hoping the answer was to use the 1.1.0 release. I'll let you know if the issues continue, but that sounds like the problem.
Thanks again, -Dave
Charles Marcus wrote:
On 6/18/2008, David Halik (dhalik@jla.rutgers.edu) wrote:
- Linux workstations running Fedora 8/9 i386 and a locally called Dovecot 1.0.14
- NFS'd homedir with Maildir setup
- NFS is on Solaris 9 sparcv9 (64bit) running Dovecot 1.0.14
NFS is only fully supported on 1.1+. This is why you're having trouble.
1.1rc10 is the latest, and the stable release is imminent, so you should have no trouble moving to it.
--
David Halik System Administrator OIT-CSS Rutgers University dhalik@jla.rutgers.edu
Gee, I've been running for a year now, albeit in an NFS environment where there are only four machines, 3 AIX (A master where the files are resident and 2 other machines as NFS clients...a mailing list server (which can write heavily to the mounts) and a login server (which writes lightly if at all)) and one Linux Fedora (reading only). I only use fcntl locks and no caching on the clients. Knock on wood, but we've had no corruption
Charles Marcus wrote:
On 6/18/2008, David Halik (dhalik@jla.rutgers.edu) wrote:
- Linux workstations running Fedora 8/9 i386 and a locally called Dovecot 1.0.14
- NFS'd homedir with Maildir setup
- NFS is on Solaris 9 sparcv9 (64bit) running Dovecot 1.0.14
NFS is only fully supported on 1.1+. This is why you're having trouble.
1.1rc10 is the latest, and the stable release is imminent, so you should have no trouble moving to it.
--
Stewart Dean, Unix System Admin, Henderson Computer Resources Center of Bard College, Annandale-on-Hudson, New York 12504 sdean@bard.edu voice: 845-758-7475, fax: 845-758-7035
When I opened your message, before I could even read it, NFS failed and corrupted everything. Shades of Shroedinger's Cat!
Just kidding (I hope) :)
Charles Marcus wrote:
On 6/18/2008, Stewart Dean (sdean@bard.edu) wrote:
Gee, I've been running for a year now,
Note I said *fully* supported.
Specifically - Timo recommends to use 1.1 if you're using NFS... but by all means, do what ever you like... :)
--
Stewart Dean, Unix System Admin, Henderson Computer Resources Center of Bard College, Annandale-on-Hudson, New York 12504 sdean@bard.edu voice: 845-758-7475, fax: 845-758-7035
On Jun 18, 2008, at 9:12 PM, David Halik wrote:
Now this setup is just a test example and not exactly what we'll be
running in production, but it tipped up the problem either way.
Since the index is shared by both the Linux i386 machine and the
sparc64 Solaris machine, if mail is accessed from both, lets say
with Pine for example, the index becomes corrupted and breaks. As
long as only one architecture only ever touches it there are no
issues.
It should only log "architecture changed" and rebuild the index. I did
some fixed related to this at some point though, might have been only
for v1.1.
I'm assuming this is an endian issue, which would make the most
sense. Is there a way around this with flags or server options? Is
this something that has maybe been addressed in 1.1.0?
No. I once thought about making it endianess-independent, but rejected
the idea soon after I figured out that it would unnecessarily slow
down nearly-100% of users and the code would become really ugly.
Although this reminds me .. dbox does rely on indexes more than mbox/
maildir. Perhaps there should be code that attempts to preserve and
convert the important data (flags, keywords) from different endianess
indexes. But this still wouldn't make it possible to quickly switch
between architectures.
One thing you could do is to specify different index files paths for
your installations. For example for sparc64:
mail_location = maildir:~/Maildir:INDEX=~/Maildir/sparc64
I just reproduced the environment and the index corrupted immediately across NFS because of the endian issue.
Jun 25 11:53:34 host IMAP(user): : Rebuilding index file /dovecot-index/index/user/.INBOX/dovecot.index: CPU architecture changed Jun 25 11:53:35 host IMAP(user): : Corrupted index cache file /dovecot-index/index/user/.INBOX/dovecot.index.cache: field header points outside file
This was starting from a clean index, first opening pine on the NFS Solaris 9 sparc machine, and then at the same time opening pine on my Fedora 9 i386 workstation.
I'm going to try the idea of splitting the indexes into two different architectures, but I'm worried that this will not be feasible when we try to scale to our 80,000 users.
By the way, I don't think this is related to the corruption, but we also have tons of these in the logs:
Jun 25 11:52:32 host IMAP(user): : Created dotlock file's timestamp is different than current time (1214409234 vs 1214409152): /dovecot-index/control/user/.INBOX/dovecot-uidlist Jun 25 11:52:32 host IMAP(user): : Created dotlock file's timestamp is different than current time (1214409235 vs 1214409152): /dovecot-index/control/user/.INBOX/dovecot-uidlist
Timo Sirainen wrote:
On Jun 18, 2008, at 9:12 PM, David Halik wrote:
Now this setup is just a test example and not exactly what we'll be running in production, but it tipped up the problem either way. Since the index is shared by both the Linux i386 machine and the sparc64 Solaris machine, if mail is accessed from both, lets say with Pine for example, the index becomes corrupted and breaks. As long as only one architecture only ever touches it there are no issues.
It should only log "architecture changed" and rebuild the index. I did some fixed related to this at some point though, might have been only for v1.1.
I'm assuming this is an endian issue, which would make the most sense. Is there a way around this with flags or server options? Is this something that has maybe been addressed in 1.1.0?
No. I once thought about making it endianess-independent, but rejected the idea soon after I figured out that it would unnecessarily slow down nearly-100% of users and the code would become really ugly.
Although this reminds me .. dbox does rely on indexes more than mbox/maildir. Perhaps there should be code that attempts to preserve and convert the important data (flags, keywords) from different endianess indexes. But this still wouldn't make it possible to quickly switch between architectures.
One thing you could do is to specify different index files paths for your installations. For example for sparc64:
mail_location = maildir:~/Maildir:INDEX=~/Maildir/sparc64
--
David Halik System Administrator OIT-CSS Rutgers University dhalik@jla.rutgers.edu
On Wed, 2008-06-25 at 12:00 -0400, David Halik wrote:
I just reproduced the environment and the index corrupted immediately across NFS because of the endian issue.
Jun 25 11:53:34 host IMAP(user): : Rebuilding index file /dovecot-index/index/user/.INBOX/dovecot.index: CPU architecture changed Jun 25 11:53:35 host IMAP(user): : Corrupted index cache file /dovecot-index/index/user/.INBOX/dovecot.index.cache: field header points outside file
I'll check later if I can reproduce this.
This was starting from a clean index, first opening pine on the NFS Solaris 9 sparc machine, and then at the same time opening pine on my Fedora 9 i386 workstation.
Why does it matter where you run Pine? Does it directly execute Dovecot on the local machine instead of connecting via TCP?
I'm going to try the idea of splitting the indexes into two different architectures, but I'm worried that this will not be feasible when we try to scale to our 80,000 users.
I'd suggest not running Dovecot on different architectures. Like if you're on a non-x86 make it connect via TCP to a x86 Dovecot server.
By the way, I don't think this is related to the corruption, but we also have tons of these in the logs:
Jun 25 11:52:32 host IMAP(user): : Created dotlock file's timestamp is different than current time (1214409234 vs 1214409152): /dovecot-index/control/user/.INBOX/dovecot-uidlist Jun 25 11:52:32 host IMAP(user): : Created dotlock file's timestamp is different than current time (1214409235 vs 1214409152): /dovecot-index/control/user/.INBOX/dovecot-uidlist
Dovecot really wants that clocks are synchronized between the NFS clients and the server. If the clock difference is more than 1 second, you'll get problems.
This was starting from a clean index, first opening pine on the NFS Solaris 9 sparc machine, and then at the same time opening pine on my Fedora 9 i386 workstation.
Why does it matter where you run Pine? Does it directly execute Dovecot on the local machine instead of connecting via TCP?
Correct. We have dovecot executing locally in each instance, with the index being shared. I'll try the TCP method and get back to you. By the way, the only reason I'm specifically doing it this way to test out what might possibly happen to our user group.
We have approximately 50,000 student accounts, and 20,000 staff accounts that all access mail in multiple fashions. We want to be able to roll out dovecot everywhere, but to do this it has to be resiliant enough to handle multiple instances of dovecot on multiple architectures. For example, a student logs into a webmail machine (sparc) and then ssh's into a linux frontend server and opens pine at the same time. This scenerio isn't likely to happen, but it could. We're just trying to cover all possibilities. Hence why I we're running the local dovecot/pine and the server side dovecot/pine... trying to see how it holds up.
So far it's been great minus the endianess issue. By the way, we're trying out seperating the index by arch and it's working pretty good right now. The only concern is how it's going to scale with regards to disk usage if we have double the number of indexs per account. We figure max of 10MB per index multiplied by 2, multiplied by 70,000... not a small number at all, but that's for us to worry about. ;) Of course, that is a worse case scenerio.
I'd suggest not running Dovecot on different architectures. Like if you're on a non-x86 make it connect via TCP to a x86 Dovecot server.
I'm going to try that out and get back to you.
By the way, I don't think this is related to the corruption, but we also have tons of these in the logs:
Jun 25 11:52:32 host IMAP(user): : Created dotlock file's timestamp is different than current time (1214409234 vs 1214409152): /dovecot-index/control/user/.INBOX/dovecot-uidlist Jun 25 11:52:32 host IMAP(user): : Created dotlock file's timestamp is different than current time (1214409235 vs 1214409152): /dovecot-index/control/user/.INBOX/dovecot-uidlist
Dovecot really wants that clocks are synchronized between the NFS clients and the server. If the clock difference is more than 1 second, you'll get problems.
I figured. Looks like we need to be a little more strict with ntp. ;)
participants (4)
-
Charles Marcus
-
David Halik
-
Stewart Dean
-
Timo Sirainen