On 6/10/2011 11:24 PM, Aliet Santiesteban Sifontes wrote:
Hello list, we continue our tests using Dovecot on a RHEL 6.1 Cluster Backend with GFS2, also we are using dovecot as a Director for user node persistence, everything was ok until we started stress testing the solution with imaptest, we had many deadlocks, cluster filesystems corruptions and hangs, specially in index filesystem, we have configured the backend as if they were on a NFS like setup but this seems not to work at least on GFS2 on rhel 6.1.
Actual _filesystem_ corruption is typically unrelated to user space applications. You should be looking at a lower level for the cause, i.e. kernel, device driver, hardware, etc. Please post details of your shared storage hardware environment, including HBAs, SAN array brand/type, if you're using GFS2 over DRBD, etc.
We have a two node cluster sharing two GFS2 filesystem
- Index GFS2 filesystem to store users indexes
- Mailbox data on a GFS2 filesystem
Experience of many users has shown that neither popular cluster filesystems such as GFS2/OCFS, nor NFS, handle high metadata/IOPS workloads very well, especially those that make heavy use of locking.
The specific configs for NFS or cluster filesystem we used:
mmap_disable = yes mail_fsync = always mail_nfs_storage = yes mail_nfs_index = yes fsync_disable=no lock_method = fcntl
mail location :
mail_location = mdbox:/var/vmail/%d/%3n/%n/mdbox:INDEX=/var/indexes/%d/%3n/%n
For a Dovecot cluster using shared storage, you are probably better off using a mailbox format for which indexes are independent of mailbox files and are automatically [re]generated if absent.
Try using mbox or maildir and store indexes on local node disk/SSD instead of on the cluster filesystem. Only store the mailboxes on the cluster filesystem. If for any reason a user login gets bumped to a node lacking the index files they're automatically rebuilt.
Since dbox indexes aren't automatically generated if missing you can't do what I describe above with dbox storage. Given the limitations of cluster filesystem (and NFS) metadata IOPS and locking, you'll likely achieve best performance and stability using local disk index files and mbox format mailboxes on GFS2. Maildir format works in this setup as well, but the metadata load on the cluster filesystem is much higher, and thus peak performance will typically be lower.
-- Stan