On 1/19/2012 6:13 PM, Timo Sirainen wrote:
On 20.1.2012, at 1.51, Stan Hoeppner wrote:
I spent a decent amount of time last night researching the NFS cache issue. It seems there is no way to completely disable NFS client caching (in lie of rewriting the code oneself--a daunting tak), which would seem to be the real solution to the mdbox index corruption problem.
So I went looking for alternatives and came up with the idea above. Obviously it's far from an optimal solution and introduces some limitations, but I thought it was worth tossing out for discussion.
I spent months looking into NFS related issues. I read through Linux and FreeBSD kernel source codes to figure out if there's something I could do to avoid the problems I see. I sent some patches to try to improve things, which of course didn't get accepted (some alternative ways might have been, but it would have required much more work from my part). The mail_nfs_* settings are the result of what I found out. They don't fully work, so I gave up.
Yeah, I recall some of your posts from that time, and your frustration. If an NFS config option existed to simply turn off the NFS client caching, would that resolve most/all of the remaining issues? Or is the problem more complex than just the file caching? I ask as it would seem creating such a Boolean NFS config option should be simple to implement. If the devs could be convinced of the need for it.
Timo, it seems that when you designed mdbox you didn't have NFS based clusters in mind. Do you consider mdbox simply not suitable for such an NFS cluster deployment? If one has no choice but an NFS cluster architecture, what Dovecot mailbox format do you recommend? Stick with maildir?
In the typical random-access NFS setup I don't consider any of Dovecot's formats suitable. Not maildir, not dbox. Perhaps in future I can redesign everything in a way that just happens to work well with all kinds of NFS setups, but I don't really hold a lot of hope for that. It seems that either you'll get bad performance (I'm not really interested in making Dovecot do that) or you'll use such a setup where you get good performance by avoiding the NFS problems.
There are several huge Dovecot+NFS setups. They use director. It works well enough (and with the recent fixes, I'd hope perfectly).
Are any of these huge setups using mdbox? Or does it make a difference? I.e. Indexes are indexes whether they be maildir or mdbox. Would Director alone allow the OP to avoid the cache corruption issues discussed in this thread? Or would there still be problems due to the split LDA setup?
In this case the OP has Netapp storage. Netapp units support both NFS exports as well as iSCSI LUNs. If the OP could utilize iSCSI instead of NFS, switching to GFS2 or OCFS, do you see these cluster filesystem as preferable for mdbox?
I don't have personal experience with cluster filesystems in recent years (other than glusterfs, which had some problems, but most(/all?) were fixed already or are available from their commercial support..). Based on what I've heard, I'm guessing they work better than random-access-NFS, but even if there are no actual corruption problems, it sounds like their performance isn't very good.
So would an ideal long term solution to indexes in a cluster (NFS or clusterFS) environment be something like Dovecot's own index metadata broker daemon/lock manager that controls access to the files/indexes? Either a distributed token based architecture, or maybe something 'simple' such as a master node which all others send index updates to with the master performing the actual writes to the files, similar to a database architecture? The former likely being more difficult to implement, the latter having potential scalability and SPOF issues.
Or is the percentage of Dovecot cluster deployments so small that it's difficult to justify the development investment for such a thing?
Thanks Timo.
-- Stan