[Dovecot] HA Dovecot Config?
Hi!
I'm currently running Dovecot 1.1.8 on a HP server using CentOS 5.2 as my IMAP server, receiving mail from postfix, and also using squirrelmail as the frontend.
As I look at upgrading the mail server, I'd like to change to a higher availability configuration (where the server can fail and I don't have to reconfig my imap users).
For the SMTP that's easy, because I can use multiple MX records, and I can redirect a port forward from one server to another. But IMAP doesn't have the same functionality, because it's the storage that matters.
What's the "best" way to do that? Clustered servers using a SAN? NAS? some sort of appliance in front? Suggestions?
Thanks!
Rick
Rick Steeves http://www.sinister.net
In reality nothing is more damaging to the adventurous spirit within a man than a secure future - Alexander Supertramp
Quoting dovecot@corwyn.net:
As I look at upgrading the mail server, I'd like to change to a
higher availability configuration (where the server can fail and I
don't have to reconfig my imap users).
Sounds like a great plan!
For the SMTP that's easy, because I can use multiple MX records, and
I can redirect a port forward from one server to another. But IMAP
doesn't have the same functionality, because it's the storage that
matters.
True for SMTP as long as you handle the SSL certificate issue, or don't use encryption.
For IMAP to be truely HA, you will need shared storage of some sort.
What's the "best" way to do that? Clustered servers using a SAN?
NAS? some sort of appliance in front? Suggestions?
There is no single "best way" since it will depend on your budget, skills, etc.
Certainly a SAN is one way to go, and allows for possibly active-active setups (depending on file system) and great flexibility.
You could also try that with NAS if you are careful enough. SAN is more complex to setup than a NAS, but NAS is harder to setup correctly for dovecot than a SAN would be, so flip a coin there.
You can "emulate" a SAN with something like DRBD if budget doesn't allow a real SAN (that is what I do).
Or you could do multi-attached active-passive disk systems (external disk tray is physically connected to 2 machines in active-passive setup).
Which to use depends on knowledge/skill, costs/budget, type of cluster/failover needed, vendor support if that matters to you, etc.
I setup mine as a pair of redundant front-end firewalls (linux heartbeat) which connect to a trio of Red Hat Cluster Suite machines using DRBD+GFS (two nodes do DRBD+GFS and handle SMTP+POP3+IMAP4, while the third node does _NOT_ do DRBD+GFS, and simply does the webmail interface).
Thanks!
Rick
-- Eric Rostetter The Department of Physics The University of Texas at Austin
Go Longhorns!
Quoting "Seth Mattinen" sethm@rollernet.us:
Eric Rostetter wrote:
For IMAP to be truely HA, you will need shared storage of some sort.
You can "emulate" a SAN with something like DRBD if budget doesn't allow a real SAN (that is what I do).
I can vouch for DRBD too. It works quite well.
~Seth
Anyone used FileReplicationPro? I'm more interested in low
bandwidth, 'cheaper', replication.
Rick
Rick Romero wrote:
Quoting "Seth Mattinen" sethm@rollernet.us:
Eric Rostetter wrote:
For IMAP to be truely HA, you will need shared storage of some sort.
You can "emulate" a SAN with something like DRBD if budget doesn't allow a real SAN (that is what I do).
I can vouch for DRBD too. It works quite well.
~Seth
Anyone used FileReplicationPro? I'm more interested in low bandwidth, 'cheaper', replication.
Rick
If data consistency isn't a must, you can always perform timed rsyncs.
R's,
Hugo Monteiro.
-- ci.fct.unl.pt:~# cat .signature
Hugo Monteiro Email : hugo.monteiro@fct.unl.pt Telefone : +351 212948300 Ext.15307 Web : http://hmonteiro.net
Centro de Informática Faculdade de Ciências e Tecnologia da Universidade Nova de Lisboa Quinta da Torre 2829-516 Caparica Portugal Telefone: +351 212948596 Fax: +351 212948548 www.ci.fct.unl.pt apoio@fct.unl.pt
ci.fct.unl.pt:~# _
Quoting Hugo Monteiro hugo.monteiro@fct.unl.pt:
Anyone used FileReplicationPro? I'm more interested in low
bandwidth, 'cheaper', replication.Rick
If data consistency isn't a must, you can always perform timed rsyncs.
Yes, but if there is a lot of data, FileReplicationPro is more efficient than an rsync. Of course, rsync is cheaper... ;)
R's,
Hugo Monteiro.
-- Eric Rostetter The Department of Physics The University of Texas at Austin
Go Longhorns!
-------- Original-Nachricht --------
Datum: Wed, 21 Oct 2009 07:42:14 -0500 Von: "Rick Romero" rick@havokmon.com An: "Dovecot Mailing List" dovecot@dovecot.org, "Seth Mattinen" sethm@rollernet.us CC: dovecot@dovecot.org Betreff: Re: [Dovecot] HA Dovecot Config?
Quoting "Seth Mattinen" sethm@rollernet.us:
Eric Rostetter wrote:
For IMAP to be truely HA, you will need shared storage of some sort.
You can "emulate" a SAN with something like DRBD if budget doesn't
allow a
real SAN (that is what I do).
I can vouch for DRBD too. It works quite well.
~Seth
Anyone used FileReplicationPro? I'm more interested in low
bandwidth, 'cheaper', replication.
I have never used FileReplicationPro but looking at what it offers it reminds me of GlusterFS. I use GlusterFS for all www data of the domains I host. I don't use jet GlusterFS for IMAP/POP storage for all domains I host. Only a small subset of the domains I host have GlusterFS for IMAP/POP but so far it works without issues and I will soon or later migrate the other domains to be on GlusterFS as well.
The setup of GlusterFS is ultra easy (compared to other clustering FS solution I have seen) and it offers some very nice functions.
In the past I have burned my fingers with older GlusterFS releases when I have tried to use it as storage for IMAP/POP but the later 2.0.x releases of GlusterFS are more stable.
Rick
Steve
Neu: GMX DSL bis 50.000 kBit/s und 200,- Euro Startguthaben! http://portal.gmx.net/de/go/dsl02
Steve wrote:
I have never used FileReplicationPro but looking at what it offers it reminds me of GlusterFS. I use GlusterFS for all www data of the domains I host. I don't use jet GlusterFS for IMAP/POP storage for all domains I host. Only a small subset of the domains I host have GlusterFS for IMAP/POP but so far it works without issues and I will soon or later migrate the other domains to be on GlusterFS as well.
The setup of GlusterFS is ultra easy (compared to other clustering FS solution I have seen) and it offers some very nice functions.
In the past I have burned my fingers with older GlusterFS releases when I have tried to use it as storage for IMAP/POP but the later 2.0.x releases of GlusterFS are more stable.
The thread moved on and no one seemed to bite, but I also have watched glusterfs for a long while now and been very attracted by the basic principle. I would be very interested to hear more about how it's worked out for you? Got any benchmarks, perhaps comparing against local storage and vs NFS?
It seems intuitively very appealing to use something heavily distributed in the style of gluster/mogile for the file data, and something fast (like local storage) for the indexes. Perhaps even dovecot proxying could be used to help force users onto a persistent server to avoid re-creating index data...
For smaller setups it would be appealing to find something simpler than DRBD for a two server active/active setup (eg both servers acting as both storage & imap and failover for each other)
Cheers
Ed W
-------- Original-Nachricht --------
Datum: Thu, 22 Oct 2009 12:57:56 +0100 Von: Ed W lists@wildgooses.com An: Dovecot Mailing List dovecot@dovecot.org Betreff: Re: [Dovecot] HA Dovecot Config?
Steve wrote:
Hallo Ed,
I have never used FileReplicationPro but looking at what it offers it reminds me of GlusterFS. I use GlusterFS for all www data of the domains I host. I don't use jet GlusterFS for IMAP/POP storage for all domains I host. Only a small subset of the domains I host have GlusterFS for IMAP/POP but so far it works without issues and I will soon or later migrate the other domains to be on GlusterFS as well.
The setup of GlusterFS is ultra easy (compared to other clustering FS solution I have seen) and it offers some very nice functions.
In the past I have burned my fingers with older GlusterFS releases when I have tried to use it as storage for IMAP/POP but the later 2.0.x releases of GlusterFS are more stable.
The thread moved on and no one seemed to bite, but I also have watched glusterfs for a long while now and been very attracted by the basic principle. I would be very interested to hear more about how it's worked out for you?
I use GlusterFS since long time. For mail hosting I waited for release 2 and then when it was out I switched all mail domains to use GlusterFS 2.0.1. It was a ultra big failure for me. The process took so much CPU that I could barely run anything else on the system and stability was very, very bad. That forced me to switch back to NFS for all the domains. Then later around 2.0.4 I looked again at it and things where more stable. I started then with a bunch of domains to run on top of GlusterFS 2.0.4 and moved then to GlusterFS GIT and somewhere around 2.0.7 the GIT version broke horribly in my setup that I switched to 2.0.7. That has been some weeks ago and since then I run around 1/3 of my domains on top of GlusterFS 2.0.7 with two active nodes using server side replicate, io-threads, write-behind and io-cache. On the client side I have no performance translators or anything such. Just bare client. I used server side replication because I wanted the shared storage to behave like a SAN and not use the server part as dumb bricks where the client is responsible for the replication. So far both nodes have 2 x 1TB disks in RAID 1 mode and those disks are then exported as a GlusterFS brick doing replicate. Setup in Dovecot is +/- like you would do if you would use NFS.
Got any benchmarks, perhaps comparing against local storage and vs NFS?
I have benchmarked but have nothing made to show to others. In general I can say that using GlusterFS on gigabit network is +/- 1/3 to 1/2 of the raw disk speed. If you add performance translators then the speed is somewhere between 1/2 to 1/1 of raw disk speed (depending what block size I use and depending if I use Booster or not). With the performance translators you can easy saturate a gigabit connection. Have not tried to use anything faster then gigabit. Off course local storage is faster since I don't have that theoretical 125MB/s limit I have when using gigabit. The newer releases of GlusterFS are comparable to NFS in terms of speed. In terms of CPU usage GlusterFS is way behind NFS and local storage. In terms of flexibility GlusterFS is better then anything else.
It seems intuitively very appealing to use something heavily distributed in the style of gluster/mogile for the file data, and something fast (like local storage) for the indexes. Perhaps even dovecot proxying could be used to help force users onto a persistent server to avoid re-creating index data...
Have never tried MoglieFS. What I don't like about it is that it uses HTTP for transport and operations.
For smaller setups it would be appealing to find something simpler than DRBD for a two server active/active setup (eg both servers acting as both storage & imap and failover for each other)
That is easy done with GlusterFS. Very easy.
Cheers
Ed W
// Steve
GRATIS für alle GMX-Mitglieder: Die maxdome Movie-FLAT! Jetzt freischalten unter http://portal.gmx.net/de/go/maxdome01
This is veering a bit OT, hence the top-post, but it looks like another HA option may be available in a few months:
http://lists.freebsd.org/pipermail/freebsd-announce/2009-October/001279.html
In short, you can stack any geom-aware FS on top of this. Combined with CARP, you've got a decent/simple/cheap option.
Pawel does really good work, I look forward to playing around with this.
Charles
Charles Sprickman NetEng/SysAdmin Bway.net - New York's Best Internet - www.bway.net spork@bway.net - 212.655.9344
On Thu, 22 Oct 2009, Steve wrote:
-------- Original-Nachricht --------
Datum: Thu, 22 Oct 2009 12:57:56 +0100 Von: Ed W lists@wildgooses.com An: Dovecot Mailing List dovecot@dovecot.org Betreff: Re: [Dovecot] HA Dovecot Config?
Steve wrote:
Hallo Ed,
I have never used FileReplicationPro but looking at what it offers it reminds me of GlusterFS. I use GlusterFS for all www data of the domains I host. I don't use jet GlusterFS for IMAP/POP storage for all domains I host. Only a small subset of the domains I host have GlusterFS for IMAP/POP but so far it works without issues and I will soon or later migrate the other domains to be on GlusterFS as well.
The setup of GlusterFS is ultra easy (compared to other clustering FS solution I have seen) and it offers some very nice functions.
In the past I have burned my fingers with older GlusterFS releases when I have tried to use it as storage for IMAP/POP but the later 2.0.x releases of GlusterFS are more stable.
The thread moved on and no one seemed to bite, but I also have watched glusterfs for a long while now and been very attracted by the basic principle. I would be very interested to hear more about how it's worked out for you?
I use GlusterFS since long time. For mail hosting I waited for release 2 and then when it was out I switched all mail domains to use GlusterFS 2.0.1. It was a ultra big failure for me. The process took so much CPU that I could barely run anything else on the system and stability was very, very bad. That forced me to switch back to NFS for all the domains. Then later around 2.0.4 I looked again at it and things where more stable. I started then with a bunch of domains to run on top of GlusterFS 2.0.4 and moved then to GlusterFS GIT and somewhere around 2.0.7 the GIT version broke horribly in my setup that I switched to 2.0.7. That has been some weeks ago and since then I run around 1/3 of my domains on top of GlusterFS 2.0.7 with two active nodes using server side replicate, io-threads, write-behind and io-cache. On the client side I have no performance translators or anything such. Just bare client. I used server side replication because I wanted the shared storage to behave like a SAN and not use the server part as dumb bricks where the client is responsible for the replication. So far both nodes have 2 x 1TB disks in RAID 1 mode and those disks are then exported as a GlusterFS brick doing replicate. Setup in Dovecot is +/- like you would do if you would use NFS.
Got any benchmarks, perhaps comparing against local storage and vs NFS?
I have benchmarked but have nothing made to show to others. In general I can say that using GlusterFS on gigabit network is +/- 1/3 to 1/2 of the raw disk speed. If you add performance translators then the speed is somewhere between 1/2 to 1/1 of raw disk speed (depending what block size I use and depending if I use Booster or not). With the performance translators you can easy saturate a gigabit connection. Have not tried to use anything faster then gigabit. Off course local storage is faster since I don't have that theoretical 125MB/s limit I have when using gigabit. The newer releases of GlusterFS are comparable to NFS in terms of speed. In terms of CPU usage GlusterFS is way behind NFS and local storage. In terms of flexibility GlusterFS is better then anything else.
It seems intuitively very appealing to use something heavily distributed in the style of gluster/mogile for the file data, and something fast (like local storage) for the indexes. Perhaps even dovecot proxying could be used to help force users onto a persistent server to avoid re-creating index data...
Have never tried MoglieFS. What I don't like about it is that it uses HTTP for transport and operations.
For smaller setups it would be appealing to find something simpler than DRBD for a two server active/active setup (eg both servers acting as both storage & imap and failover for each other)
That is easy done with GlusterFS. Very easy.
Cheers
Ed W
// Steve
GRATIS f?r alle GMX-Mitglieder: Die maxdome Movie-FLAT! Jetzt freischalten unter http://portal.gmx.net/de/go/maxdome01
On Thu, Oct 22, 2009 at 12:57:56PM +0100, Ed W wrote:
Steve wrote:
...
The thread moved on and no one seemed to bite, but I also have watched
glusterfs for a long while now and been very attracted by the basic
principle. I would be very interested to hear more about how it's worked
out for you? Got any benchmarks, perhaps comparing against local
storage and vs NFS?
I'd also recommend testing DRBD
I tested and ran a glusterfs cluster on 3 production servers for several months, however with only a small load, nothing substantial. Through testing and production, I didn't run into a single issue, stability or otherwise.
However, having said that, and as long as we're talking HA...
Complaints about stability are very common on the glusterfs mailing lists. I also have issue with the fuse layer, it has always been unstable in my experiences with it. Because of this, I dropped the idea of a glusterfs cluster. IMO, it isn't quite production ready.
Maybe that's just my pain threshold though, and it doesn't mean I find glusterfs unusable either. But, mail users are finicky, and waste enough of my time already.
I am actively pursuing a DRBD cluster now. Benchmarks on DRBD + OCFS2 showed similar throughput, and the project appears to be more stable.
Anthony
Quoting Rick Romero rick@havokmon.com:
Anyone used FileReplicationPro? I'm more interested in low
bandwidth, 'cheaper', replication.
Might work for an active-failover setup, but since I use active-active I need something like DRBD instead.
Personally I wouldn't trust it for cluster type situations, and I kind of doubt the authors would either though I could be wrong.
Certainly a great choice for backups though... Depending on your needs, it might work for a failover node...
Rick
-- Eric Rostetter The Department of Physics The University of Texas at Austin
This message is provided "AS IS" without warranty of any kind, either expressed or implied. Use this message at your own risk.
Quoting "Eric Jon Rostetter" eric.rostetter@physics.utexas.edu:
Quoting Rick Romero rick@havokmon.com:
Anyone used FileReplicationPro? I'm more interested in low
bandwidth, 'cheaper', replication.Might work for an active-failover setup, but since I use active-active I need something like DRBD instead.
Personally I wouldn't trust it for cluster type situations, and I kind of doubt the authors would either though I could be wrong.
Certainly a great choice for backups though... Depending on your needs, it might work for a failover node...
They claim to be real-time, and with kernel hooks into reads/writes,
it seems promising to run on top of the OS..
http://www.filereplicationpro.com/sol-branch-office-sync.cfm
Maybe I should voice my assumption that a Maildir mailbox would only
be accessed from one server at a time, not both, and Dovecot indexes
would be local and not replicated. And I suppose I'm assuming small
files - which is probably the biggest hole in this active/active theory.
Rick
Quoting Rick Romero rick@havokmon.com:
They claim to be real-time, and with kernel hooks into reads/writes,
it seems promising to run on top of the OS..
Yes, but that doesn't mean it will work in the cluster environment...
It also says things like:
There are different types of file locks. In addition to these differences, each OS has its own set of rules regarding file locks as well. FRP's replication of locked files will vary in relation to the way the OS treats these types. However, one rule is consistent throughout all operating systems in that exclusive file locks will prevent FRP from replicating data.
See that last line there??? Want to bet something in the mail setup (MTA, dovecot, system jobs like backup, etc) put an exclusive lock on files from time to time?
Again, a fine backup system, and depending on your needs it might be okay for a failover setup, but not for an active-active setup. For that you need a lock manager, which they promise "in the future" but don't deliver yet...
Rick
-- Eric Rostetter The Department of Physics The University of Texas at Austin
Go Longhorns!
Quoting "Eric Rostetter" rostetter@mail.utexas.edu:
Quoting Rick Romero rick@havokmon.com:
They claim to be real-time, and with kernel hooks into
reads/writes, it seems promising to run on top of the OS..Yes, but that doesn't mean it will work in the cluster environment...
It also says things like:
There are different types of file locks. In addition to these differences, each OS has its own set of rules regarding file locks as well. FRP's replication of locked files will vary in relation to the way the OS treats these types. However, one rule is consistent throughout all operating systems in that exclusive file locks will prevent FRP from replicating data.
See that last line there??? Want to bet something in the mail setup (MTA, dovecot, system jobs like backup, etc) put an exclusive lock on files from time to time?
Again, a fine backup system, and depending on your needs it might be okay for a failover setup, but not for an active-active setup. For that you need a lock manager, which they promise "in the future" but don't deliver yet...
I thought the entire reason (ok not the _entire_ reason but..) for
Maildir was to avoid locks and allow easy concurrent access to a
mailbox through many different applications.
From DJB's page: http://cr.yp.to/proto/maildir.html
Why should I use maildir? Two words: no locks
So while I do agree that file locking is a possible problem in an
active/active setup with FRP, I think it's possible - as long as the
admin is aware of the risks. So for backup purposes, to avoid locks
for example, use ZFS snapshots.
I'm not even a user of FRP at this point, but it's something that
intrigues me for active/active because of it's OS independence. I
want ZFS. I really enjoy ZFS. I already have active/passive with zfs
sends, I want to step it up :)
Yes, there are definitely things to be aware of. I'm not sure if this
is the place to hash it out, but I think that while it's not a cheap
solution it may fit MY environment - and possibly others who use
Maildir.
I guess everyone is as cheap as I am and hasn't set it up yet :)
Rick
On Wed, 2009-10-21 at 15:35 -0500, Rick Romero wrote:
I thought the entire reason (ok not the _entire_ reason but..) for
Maildir was to avoid locks and allow easy concurrent access to a
mailbox through many different applications.From DJB's page: http://cr.yp.to/proto/maildir.html
Why should I use maildir? Two words: no locks
Yeah, right. http://wiki.dovecot.org/MailboxFormat/Maildir#Locking
Quoting Rick Romero rick@havokmon.com:
See that last line there??? Want to bet something in the mail setup (MTA, dovecot, system jobs like backup, etc) put an exclusive lock on files from time to time?
I thought the entire reason (ok not the _entire_ reason but..) for
Maildir was to avoid locks and allow easy concurrent access to a
mailbox through many different applications.
Okay, so if you are real careful (use maildir instead of some other format like mbox, don't use it for your dovecot index/control files, don't do native backups but use a snapshot, don't involve any other programs which might introduce locks, etc. then you can probably get away with it. But I still wouldn't bet on it.
So while I do agree that file locking is a possible problem in an
active/active setup with FRP, I think it's possible - as long as the
admin is aware of the risks.
And the ways to avoid locking... And assuming locking is the only issue (other issues may arise depending on implementation).
Yes, there are definitely things to be aware of. I'm not sure if
this is the place to hash it out, but I think that while it's not a
cheap solution it may fit MY environment - and possibly others who
use Maildir.
OP didn't specify many details, so assumptions were made... On my system, I share mbox files and dovecot indexes, and they both having locking...
I guess everyone is as cheap as I am and hasn't set it up yet :)
Uhm, DRBD is free and this suggested alternative isn't AFAIK, so I don't think it is being cheap... I'm thinking it is more about reputation and install bases...
Rick
-- Eric Rostetter The Department of Physics The University of Texas at Austin
Go Longhorns!
participants (11)
-
Anthony
-
Charles Sprickman
-
dovecot＠corwyn.net
-
Ed W
-
Eric Jon Rostetter
-
Eric Rostetter
-
Hugo Monteiro
-
Rick Romero
-
Seth Mattinen
-
Steve
-
Timo Sirainen