Dovecot cluster using GlusterFS
Hello,
I have recently setup mailserver solution using 2-node master-master setup (mainly based on MySQL M-M replication and GlusterFS with 2 replica volume) on Ubuntu 14.04 (Dovecot 2.2.9).
Unfortunately even with shared-storage-aware setting:
mail_nfs_index = yes mail_nfs_storage = yes mail_fsync = always mmap_disable = yes
..I have hit strange issues pretty soon especially when user was manipulating same mailbox from multiple devices at the same time.
Most issues was about corrupted indexes which was solved easily by just putting them on local storage of each node:
mail_location = maildir:/srv/mail/%d/%u:INDEX=/var/lib/dovecot/index/%d/%u
But I still hit issues like this one:
dovecot: lmtp(6276, user@example.com): Error: Broken file /srv/mail/example.com/user@example.com/dovecot-uidlist line 8529: UIDs not ordered (8527 >= 8527)
Which I am not sure how serious it is or if it's possible to solve or workaround?
Anyway because of the above and high possibility of GlusterFS split-brains, I have decided to setup Dovecot Director according to the docs [1] but I have a couple of questions:
is custom monitoring still required? Poolmon [2] is 4 year old so I would suppose there's some progress since that?
it's not possible to have same backends and directors in Dovecot <2.2.17. I can backport newer Dovecot for Ubuntu Trusty, so this is not an issue, but..
documentation states that it still doesn't work for LMTP [3]? Which is probably important for my setup, because both Postfix servers are using dovecot-lmtp for mail delivery so there can be still some issues (but probably less frequent?) when both servers will deliver new mails for one user at once. So do I really have to split directors from backends?
Anyone has experience with clustered Dovecot setup? Why is Dovecot behaving so bad when it pretends to be shared storage friendly? Are these issues only specific for older Dovecot? Or is there something wrong in my architecture design?
Thanks for any help, Filip
[1] http://wiki2.dovecot.org/Director [2] https://github.com/brandond/poolmon/ [3] "LMTP however doesn't currently support mixing recipients to both being proxied and store locally."
BTW if someone is interested in SaltStack, here are Salt formulas for Dovecot + Postfix + GlusterFS + Roundcube + Mailman setup which we are using:
https://github.com/tcpcloud/salt-formula-dovecot https://github.com/tcpcloud/salt-formula-postfix https://github.com/tcpcloud/salt-formula-roundcube https://github.com/tcpcloud/salt-formula-glusterfs
Hi Filip,
On 12/05/2015 10:42 AM, Filip Pytloun wrote:
I have recently setup mailserver solution using 2-node master-master setup (mainly based on MySQL M-M replication and GlusterFS with 2 replica volume) on Ubuntu 14.04 (Dovecot 2.2.9).
that's no good idea due to different reasons - see below.
Anyway because of the above and high possibility of GlusterFS split-brains, I have decided to setup Dovecot Director according to the docs [1] but I have a couple of questions:
- is custom monitoring still required? Poolmon [2] is 4 year old so I would suppose there's some progress since that?
Using the dovecot director, poolmon is strongly recommended - see the official dovecot wiki. The tool seems to be old, but the task is the same since a couple of years ;-)
- it's not possible to have same backends and directors in Dovecot <2.2.17. I can backport newer Dovecot for Ubuntu Trusty, so this is not an issue, but..
You have to use 2 instances of dovecot when running the director and the backend on the same server. Otherwise, you have to use 2 systems.
- documentation states that it still doesn't work for LMTP [3]? Which is probably important for my setup, because both Postfix servers are using dovecot-lmtp for mail delivery so there can be still some issues (but probably less frequent?) when both servers will deliver new mails for one user at once. So do I really have to split directors from backends?
At the moment, I cannot recognize the requirement for using lmtp over the directors. When using postfix for delivering e-mails to the backend, do this directly with an corresponding MX record.
Why is Dovecot behaving so bad when it pretends to be shared storage friendly? Are these issues only specific for older Dovecot? Or is there something wrong in my architecture design?
IMHO, this is not a problem of dovecot. In general, using cluster filesystems in such a setup is no good idea. You've already mentioned all the problems of such a setup. In particular, the performance of GlusterFS is absolutely not suitable for (bigger) mailserver cluster.
Asking any search engine for "dovecot clustering" shows a lot of results for good cluster designs. In particular, a shared mailbox storage should be avoided. Here, the dovecot internal replication mechanism on the backends seems to be the best solution. The rest of the setup (e.g., clustered directors or smtp servers) is trivial. Due to the fact, that MUAs are able to use A records only and the behavior using round robin A records is catastrophically, a service IP address is recommended. Here, keepalived is your friend.
HTH.
Best regards, Gordon
Technischer Leiter & stellv. Direktor Universitätsrechenzentrum (URZ) E.-M.-Arndt-Universität Greifswald Felix-Hausdorff-Str. 12 17489 Greifswald Germany
Tel. +49 3834 86 1456 Fax. +49 3834 86 1401
Il 05.12.2015 10:42 Filip Pytloun ha scritto:
Hello,
I have recently setup mailserver solution using 2-node master-master setup (mainly based on MySQL M-M replication and GlusterFS with 2 replica volume) on Ubuntu 14.04 (Dovecot 2.2.9).
Unfortunately even with shared-storage-aware setting:
mail_nfs_index = yes mail_nfs_storage = yes mail_fsync = always mmap_disable = yes
With only these setting you don't solve the problem of shared storage.
..I have hit strange issues pretty soon especially when user was manipulating same mailbox from multiple devices at the same time.
Most issues was about corrupted indexes which was solved easily by just putting them on local storage of each node:
mail_location = maildir:/srv/mail/%d/%u:INDEX=/var/lib/dovecot/index/%d/%u
But I still hit issues like this one:
dovecot: lmtp(6276, user@example.com): Error: Broken file /srv/mail/example.com/user@example.com/dovecot-uidlist line 8529: UIDs not ordered (8527 >= 8527)
Which I am not sure how serious it is or if it's possible to solve or workaround?
You need Director for POP/IMAP and also LMTP so you can solve all "Broken file" and "corrupted indexes" problems.
Anyway because of the above and high possibility of GlusterFS split-brains, I have decided to setup Dovecot Director according to the docs [1] but I have a couple of questions:
- is custom monitoring still required? Poolmon [2] is 4 year old so I would suppose there's some progress since that?
For me poolmon works fine.
- it's not possible to have same backends and directors in Dovecot <2.2.17. I can backport newer Dovecot for Ubuntu Trusty, so this is not an issue, but..
Yes is possibile (also with < 2.2.17), create two instances, like dovecot and director, two config directory /etc/dovecot/ and /etc/director/ and bind on differents IPs.
- documentation states that it still doesn't work for LMTP [3]? Which is probably important for my setup, because both Postfix servers are using dovecot-lmtp for mail delivery so there can be still some issues (but probably less frequent?) when both servers will deliver new mails for one user at once. So do I really have to split directors from backends?
I'm running Director and backend on the same server for POP/IMAP, and in another configuration and Director for LMTP is on the same server (but with 2.2.19).
Anyone has experience with clustered Dovecot setup? Why is Dovecot behaving so bad when it pretends to be shared storage friendly? Are these issues only specific for older Dovecot? Or is there something wrong in my architecture design?
You need Director, Dovecot has not problems with shared storage, big installation are always using shared storage (like NFS).
Alessio Cecchi Postmaster AT http://www.qboxmail.it http://www.linkedin.com/in/alessice
Hello Alessio and Gordon, thank you for answers.
Dsync-based architecture looks promising, but I would preffer to stay with GlusterFS for now as I also use it as a storage for other components.
So director is the way to go, I don't want to setup more than two nodes to keep this setup as simple as possible - so I will probably update to 2.2.19 and have director and backend on the same servers (and Dovecot instance).
I asked about poolmon, because I think that Dovecot should have some internal mechanism on how to recognize broken backend by default. But if it works nicely, I am going to use it as well :-)
At the moment, I cannot recognize the requirement for using lmtp over the directors. When using postfix for delivering e-mails to the backend, do this directly with an corresponding MX record.
I have two MX records of the same weight with postfix using dovecot-lmtp for delivery. So that's why I wanted to use LMTP over directors. Using lower weight for second MX is an option, but not truly master-master setup :-)
Filip
On 2015/12/06 02:31, Alessio Cecchi wrote:
Il 05.12.2015 10:42 Filip Pytloun ha scritto:
Hello,
I have recently setup mailserver solution using 2-node master-master setup (mainly based on MySQL M-M replication and GlusterFS with 2 replica volume) on Ubuntu 14.04 (Dovecot 2.2.9).
Unfortunately even with shared-storage-aware setting:
mail_nfs_index = yes mail_nfs_storage = yes mail_fsync = always mmap_disable = yes
With only these setting you don't solve the problem of shared storage.
..I have hit strange issues pretty soon especially when user was manipulating same mailbox from multiple devices at the same time.
Most issues was about corrupted indexes which was solved easily by just putting them on local storage of each node:
mail_location = maildir:/srv/mail/%d/%u:INDEX=/var/lib/dovecot/index/%d/%u
But I still hit issues like this one:
dovecot: lmtp(6276, user@example.com): Error: Broken file /srv/mail/example.com/user@example.com/dovecot-uidlist line 8529: UIDs not ordered (8527 >= 8527)
Which I am not sure how serious it is or if it's possible to solve or workaround?
You need Director for POP/IMAP and also LMTP so you can solve all "Broken file" and "corrupted indexes" problems.
Anyway because of the above and high possibility of GlusterFS split-brains, I have decided to setup Dovecot Director according to the docs [1] but I have a couple of questions:
- is custom monitoring still required? Poolmon [2] is 4 year old so I would suppose there's some progress since that?
For me poolmon works fine.
- it's not possible to have same backends and directors in Dovecot <2.2.17. I can backport newer Dovecot for Ubuntu Trusty, so this is not an issue, but..
Yes is possibile (also with < 2.2.17), create two instances, like dovecot and director, two config directory /etc/dovecot/ and /etc/director/ and bind on differents IPs.
- documentation states that it still doesn't work for LMTP [3]? Which is probably important for my setup, because both Postfix servers are using dovecot-lmtp for mail delivery so there can be still some issues (but probably less frequent?) when both servers will deliver new mails for one user at once. So do I really have to split directors from backends?
I'm running Director and backend on the same server for POP/IMAP, and in another configuration and Director for LMTP is on the same server (but with 2.2.19).
Anyone has experience with clustered Dovecot setup? Why is Dovecot behaving so bad when it pretends to be shared storage friendly? Are these issues only specific for older Dovecot? Or is there something wrong in my architecture design?
You need Director, Dovecot has not problems with shared storage, big installation are always using shared storage (like NFS).
Alessio Cecchi Postmaster AT http://www.qboxmail.it http://www.linkedin.com/in/alessice
On 12/06/2015 10:16 AM, Filip Pytloun wrote:
At the moment, I cannot recognize the requirement for using lmtp over the directors. When using postfix for delivering e-mails to the backend, do this directly with an corresponding MX record.
I have two MX records of the same weight with postfix using dovecot-lmtp for delivery. So that's why I wanted to use LMTP over directors. Using lower weight for second MX is an option, but not truly master-master setup :-)
We're using postfix for delivering to the dovecot backends via lmtp on the base of an MX record with the same weight. Here, postfix can directly access the dovecot backends without using the dovecot directors. It a really master-master setup ;-)
Best regards, Gordon
On 05 Dec 2015, at 11:42, Filip Pytloun filip@pytloun.cz wrote:
Anyway because of the above and high possibility of GlusterFS split-brains, I have decided to setup Dovecot Director according to the docs [1] but I have a couple of questions:
- is custom monitoring still required? Poolmon [2] is 4 year old so I would suppose there's some progress since that?
I think it's always going to be a separate script. In different environments people may want to do it slightly differently.
- documentation states that it still doesn't work for LMTP [3]? Which is probably important for my setup, because both Postfix servers are using dovecot-lmtp for mail delivery so there can be still some issues (but probably less frequent?) when both servers will deliver new mails for one user at once. So do I really have to split directors from backends?
You can run director and backend in the same servers, but they'd have to be in different config files (so two dovecot instances). Or you could also do it with a single instance:
[3] "LMTP however doesn't currently support mixing recipients to both being proxied and store locally."
This is a problem only if there are multiple recipients in the same LMTP session. So if you configure your MTA to restrict the recipient limit to 1, this mixing can't happen and a single Dovecot instance can work.
v2.3 will solve this problem completely.
We ran a load test using glusterfs and were able to deliver mail (I can't remember specifically how much per second, maybe 100 messages per second?) without any issues. We did use the glusterfs fuse client and not nfs, and used regular maildir. We developed a mail bot cluster that would deliver mail, and simultaneously receive and delete it with pop and IMAP and we ran into zero issues. We even had the replicas stretched between two datacenters. Not sure what the difference here is but it can be done.
On Dec 5, 2015, at 3:42 AM, Filip Pytloun filip@pytloun.cz wrote:
Hello,
I have recently setup mailserver solution using 2-node master-master setup (mainly based on MySQL M-M replication and GlusterFS with 2 replica volume) on Ubuntu 14.04 (Dovecot 2.2.9).
Unfortunately even with shared-storage-aware setting:
mail_nfs_index = yes mail_nfs_storage = yes mail_fsync = always mmap_disable = yes
..I have hit strange issues pretty soon especially when user was manipulating same mailbox from multiple devices at the same time.
Most issues was about corrupted indexes which was solved easily by just putting them on local storage of each node:
mail_location = maildir:/srv/mail/%d/%u:INDEX=/var/lib/dovecot/index/%d/%u
But I still hit issues like this one:
dovecot: lmtp(6276, user@example.com): Error: Broken file /srv/mail/example.com/user@example.com/dovecot-uidlist line 8529: UIDs not ordered (8527 >= 8527)
Which I am not sure how serious it is or if it's possible to solve or workaround?
Anyway because of the above and high possibility of GlusterFS split-brains, I have decided to setup Dovecot Director according to the docs [1] but I have a couple of questions:
is custom monitoring still required? Poolmon [2] is 4 year old so I would suppose there's some progress since that?
it's not possible to have same backends and directors in Dovecot <2.2.17. I can backport newer Dovecot for Ubuntu Trusty, so this is not an issue, but..
documentation states that it still doesn't work for LMTP [3]? Which is probably important for my setup, because both Postfix servers are using dovecot-lmtp for mail delivery so there can be still some issues (but probably less frequent?) when both servers will deliver new mails for one user at once. So do I really have to split directors from backends?
Anyone has experience with clustered Dovecot setup? Why is Dovecot behaving so bad when it pretends to be shared storage friendly? Are these issues only specific for older Dovecot? Or is there something wrong in my architecture design?
Thanks for any help, Filip
[1] http://wiki2.dovecot.org/Director [2] https://github.com/brandond/poolmon/ [3] "LMTP however doesn't currently support mixing recipients to both being proxied and store locally."
BTW if someone is interested in SaltStack, here are Salt formulas for Dovecot + Postfix + GlusterFS + Roundcube + Mailman setup which we are using:
https://github.com/tcpcloud/salt-formula-dovecot https://github.com/tcpcloud/salt-formula-postfix https://github.com/tcpcloud/salt-formula-roundcube https://github.com/tcpcloud/salt-formula-glusterfs
Am Sonntag, 6. Dezember 2015, 20:33:32 schrieb list@airstreamcomm.net:
We ran a load test using glusterfs and were able to deliver mail (I can't remember specifically how much per second, maybe 100 messages per second?) without any issues. We did use the glusterfs fuse client and not nfs, and used regular maildir. We developed a mail bot cluster that would deliver mail, and simultaneously receive and delete it with pop and IMAP and we ran into zero issues. We even had the replicas stretched between two datacenters. Not sure what the difference here is but it can be done.
On Dec 5, 2015, at 3:42 AM, Filip Pytloun filip@pytloun.cz wrote:
Hello,
I have recently setup mailserver solution using 2-node master-master setup (mainly based on MySQL M-M replication and GlusterFS with 2 replica volume) on Ubuntu 14.04 (Dovecot 2.2.9).
Unfortunately even with shared-storage-aware setting:
mail_nfs_index = yes mail_nfs_storage = yes mail_fsync = always mmap_disable = yes
..I have hit strange issues pretty soon especially when user was manipulating same mailbox from multiple devices at the same time.
Most issues was about corrupted indexes which was solved easily by just putting them on local storage of each node:
mail_location = maildir:/srv/mail/%d/%u:INDEX=/var/lib/dovecot/index/%d/%u
But I still hit issues like this one:
dovecot: lmtp(6276, user@example.com): Error: Broken file /srv/mail/example.com/user@example.com/dovecot-uidlist line 8529: UIDs not ordered (8527 >= 8527)
Which I am not sure how serious it is or if it's possible to solve or workaround?
hi,
I did experiments with glusterfs and dovecot about one year ago and ran into the same trouble. It was quite easy to corrupt users mailboxes delivering mail somultaniously on different nodes.
I tried a lot of different configurations (see NFS recomendations, ...) but nothig really solved the problem.
Also from the list I did not get any useful comments.
So I decided that plain dovecot / glusterfs is not usable.
I would be glad to hear of any other experiance and configuration tweaks.
Mit freundlichen Grüßen,
Michael Schwartzkopff
-- [*] sys4 AG
http://sys4.de, +49 (89) 30 90 46 64, +49 (162) 165 0044 Franziskanerstraße 15, 81669 München
Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263 Vorstand: Patrick Ben Koetter, Marc Schiffbauer Aufsichtsratsvorsitzender: Florian Kirstein
participants (6)
-
Alessio Cecchi
-
Filip Pytloun
-
Gordon Grubert
-
list＠airstreamcomm.net
-
Michael Schwartzkopff
-
Timo Sirainen