Dovecot cluster using GlusterFS

[list at airstreamcomm.net]

We ran a load test using glusterfs and were able to deliver mail (I can't remember specifically how much per second, maybe 100 messages per second?) without any issues.  We did use the glusterfs fuse client and not nfs, and used regular maildir.  We developed a mail bot cluster that would deliver mail, and simultaneously receive and delete it with pop and IMAP and we ran into zero issues.  We even had the replicas stretched between two datacenters.  Not sure what the difference here is but it can be done.



> On Dec 5, 2015, at 3:42 AM, Filip Pytloun <filip at pytloun.cz> wrote:
> 
> Hello,
> 
> I have recently setup mailserver solution using 2-node master-master
> setup (mainly based on MySQL M-M replication and GlusterFS with 2
> replica volume) on Ubuntu 14.04 (Dovecot 2.2.9).
> 
> Unfortunately even with shared-storage-aware setting:
> 
> mail_nfs_index = yes
> mail_nfs_storage = yes
> mail_fsync = always
> mmap_disable = yes
> 
> ..I have hit strange issues pretty soon especially when user was
> manipulating same mailbox from multiple devices at the same time.
> 
> Most issues was about corrupted indexes which was solved easily by just
> putting them on local storage of each node:
> 
> mail_location = maildir:/srv/mail/%d/%u:INDEX=/var/lib/dovecot/index/%d/%u
> 
> But I still hit issues like this one:
> 
> dovecot: lmtp(6276, user at example.com): Error: Broken file /srv/mail/example.com/user at example.com/dovecot-uidlist line 8529: UIDs not ordered (8527 >= 8527)
> 
> Which I am not sure how serious it is or if it's possible to solve or
> workaround?
> 
> Anyway because of the above and high possibility of GlusterFS
> split-brains, I have decided to setup Dovecot Director according to the
> docs [1] but I have a couple of questions:
> 
> - is custom monitoring still required? Poolmon [2] is 4 year old so I
>  would suppose there's some progress since that?
> 
> - it's not possible to have same backends and directors in Dovecot
>  <2.2.17. I can backport newer Dovecot for Ubuntu Trusty, so this is
>  not an issue, but..
> 
> - documentation states that it still doesn't work for LMTP [3]?
>  Which is probably important for my setup, because both Postfix servers
>  are using dovecot-lmtp for mail delivery so there can be still some
>  issues (but probably less frequent?) when both servers will deliver
>  new mails for one user at once.
>  So do I really have to split directors from backends?
> 
> 
> Anyone has experience with clustered Dovecot setup?
> Why is Dovecot behaving so bad when it pretends to be shared storage
> friendly? Are these issues only specific for older Dovecot?
> Or is there something wrong in my architecture design?
> 
> Thanks for any help,
> Filip
> 
> 
> ---
> [1] http://wiki2.dovecot.org/Director
> [2] https://github.com/brandond/poolmon/
> [3] "LMTP however doesn't currently support mixing recipients to both
>     being proxied and store locally."
> 
> ---
> BTW if someone is interested in SaltStack, here are Salt formulas for
> Dovecot + Postfix + GlusterFS + Roundcube + Mailman setup which we are
> using:
> 
> https://github.com/tcpcloud/salt-formula-dovecot
> https://github.com/tcpcloud/salt-formula-postfix
> https://github.com/tcpcloud/salt-formula-roundcube
> https://github.com/tcpcloud/salt-formula-glusterfs