[Dovecot] Clustering (replication and proxying) plans for the future

[Bill Boebel]

On Tue, October 24, 2006 2:31 pm, Timo Sirainen <tss at iki.fi> said:

> The replication could work simply by sending the transaction logs'
> contents to the replicatio process which passes it onto other servers,
> which finally sync their local mailboxes based on that data. Since
> Dovecot already is able to sync mailboxes based on the transaction log's
> contents this should be pretty easy to implement.
> 
> Of course the new mails' contents also have to be sent. This could be
> prioritized lower than the transaction traffic, so that each server
> always has very up-to-date view of the mailbox metadata, but not
> necessarily the contents of all the mails.
> 
> If the server finds itself in a situation that it doesn't have some
> specific mail, it'll send a request to the replication process to fetch
> it ASAP from another server. The reply will then take the highest
> priority in the queue.

Is your primary goal with this replication to add redundancy or to distribute load such as for shared mailboxes?  I've thought about the redundancy side of this a lot, but not so much about load distribution.  It sounds like you're going for load distribution, but...

If the goal is redundancy, I'd suggest that mailbox state is less important than the mail data.  Index files can be recreated from the data, and would not even be needed on the secondary server(s) unless the primary fails.  So I'd put a higher priority on getting the mail content to the secondary servers.  If the goal is to distribute load for shared mailboxes, then your prioritization makes sense.

Also, if the goal is redundancy, this design can be simplified a lot by having one master and one or more slaves.  All modify operations would go to the master, so that you do not need global locks.  Different sets of users would belong to different namespaces, so that you can have multiple replication paths in your cluster.

You'd also want an API for the replication process so that third-party applications that modify mailboxes can log those changes so that they are replicated.  And in that case, the replication process could be made generic enough so that it can be used to replicate any set of files, with or without Dovecot.

Bill