[Dovecot] Replication plans

[Troy Benjegerdes]

On Fri, May 18, 2007 at 11:41:46AM +0900, Christian Balzer wrote:
> On Thu, 17 May 2007 19:17:25 +0300 Timo Sirainen <tss at iki.fi> wrote:
> 
> > On Thu, 2007-05-17 at 10:04 -0500, Troy Benjegerdes wrote:
> > > But that's currently not *really* replicated. The real question I guess
> > > is why not use a cluster/distributed/san filesystem like AFS, GFS,
> > > Lustre, GPFS to handle the actual data, and specify that replication
> > > only works for maildir or other single file per message mailbox
> > > formats.
> > 
> > This already works, but people don't seem to want to use it. There are
> > apparently some stability, performance, complexity and whatever
> > problems. And if you're planning on replicating to a remote location far
> > away they're really slow. But I haven't personally tried any of them, so
> > all I can say is that people are interested in getting replication that
> > doesn't require clustered filesystems, so I'll build it for them. :)
> > 
> I for one would rather pay you for not re-inventing the wheel, but
> if people with actual access to funds are willing to pay you for this
> then I guess "take the money and run" is the thing to do. :-p
 
I'm going to throw out a warning that it's my feeling that replication
has ended many otherwise worthwhile projects. Once you go down that
rabbit hole, you end up finding out the hard way that you just can't
avoid the stability, performance, complexity, and whatever problems.

If you take the money and run, just be aware of the complexity and
customer expectations you are getting yourself into ;)

> Yes, all these FS based approaches currently have one or more of
> the issues Timo lists. The question of course is, will a replicated
> dovecot be less complex, slow, etc.
> For people with money, there are enterprise level replicated file
> systems and/or hardware replicated SANs (remote locations, too).
> For people w/o those funds there are the above approaches (which
> despite all their shortcomings can work, right now) and of course 
> one frequently overlooked but perfectly fitting solution, DRBD.
> For the ZFS fanbois, there ware ways to make it clustered/replicated
> as well (some storageworks add-on or geom tricks).
> 
> The point (for me) would be to not just replicate IMAP (never mind
> that most of our users use POP, me preferring not to use the 
> dovecot LDA, etc), but ALL of the services/infrastructure that make
> up a mail system. Which leads quickly back to HA/cluster/SAN/DRBD
> for me. 

I've found myself pretty much in the same "all roads lead to the
filesystem" situation. I don't want to replicate just imap, I want to
replicate the build directory with my source code, my email, and my MP3
files.

So maybe the right thing to do here is have dovecot do the locking and
proxying bit, and initially use librsync for the actual replication. The
rsync bit could be replaced with plugins for various filesystems.