[Dovecot] mmap in GFS2 on rhel 6.1

[Stan Hoeppner]

On 6/10/2011 11:24 PM, Aliet Santiesteban Sifontes wrote:
> Hello list, we continue our tests using Dovecot on a RHEL 6.1 Cluster
> Backend with GFS2, also we are using dovecot as a Director for user node
> persistence, everything was ok until we started stress testing the solution
> with imaptest, we had many deadlocks, cluster filesystems corruptions and
> hangs, specially in index filesystem, we have configured the backend as if
> they were on a NFS like setup but this seems not to work at least on GFS2 on
> rhel 6.1.

Actual _filesystem_ corruption is typically unrelated to user space
applications.  You should be looking at a lower level for the cause,
i.e. kernel, device driver, hardware, etc.  Please post details of your
shared storage hardware environment, including HBAs, SAN array
brand/type, if you're using GFS2 over DRBD, etc.

> We have a two node cluster sharing two GFS2 filesystem
> - Index GFS2 filesystem to store users indexes
> - Mailbox data on a GFS2 filesystem

Experience of many users has shown that neither popular cluster
filesystems such as GFS2/OCFS, nor NFS, handle high metadata/IOPS
workloads very well, especially those that make heavy use of locking.

> The specific configs for NFS or cluster filesystem we used:
> 
> mmap_disable = yes
> mail_fsync = always
> mail_nfs_storage = yes
> mail_nfs_index = yes
> fsync_disable=no
> lock_method = fcntl
> 
> mail location :
> 
> mail_location =
> mdbox:/var/vmail/%d/%3n/%n/mdbox:INDEX=/var/indexes/%d/%3n/%n

For a Dovecot cluster using shared storage, you are probably better off
using a mailbox format for which indexes are independent of mailbox
files and are automatically [re]generated if absent.

Try using mbox or maildir and store indexes on local node disk/SSD
instead of on the cluster filesystem.  Only store the mailboxes on the
cluster filesystem.  If for any reason a user login gets bumped to a
node lacking the index files they're automatically rebuilt.

Since dbox indexes aren't automatically generated if missing you can't
do what I describe above with dbox storage.  Given the limitations of
cluster filesystem (and NFS) metadata IOPS and locking, you'll likely
achieve best performance and stability using local disk index files and
mbox format mailboxes on GFS2.  Maildir format works in this setup as
well, but the metadata load on the cluster filesystem is much higher,
and thus peak performance will typically be lower.

-- 
Stan