[Dovecot] Replication problem I have. And how I think I can get around problem.

[Ed W]

On 03/07/2011 07:20, Peter Dolding wrote:
> Now what I need to be able to deal with the problem.

Have you considered a new Dovecot storage backend?  I have plotted some
designs on a napkin a few times.. Consider some kind of storage server
with "eventually consistent" replication capabilities.  This could be
used for the metadata storage for all the emails (ie FROM, TO, DATE,
SUBJECT and all the other non body parts you might search on)

Your replication engine can now work in conjunction with Dovecot to sync
changes between servers as quickly as possible, eg if desired implement
a two phase commit when the LDA delivers new emails, so that all storage
servers confirm they have received the new email.  You could if you wish
implement quorum support (ie some server which was offline for some mail
deliveries proxies requests to another storage block until it's caught
up syncing)

You may or may not store the message bodies and attachments with the
mail metadata.  I can see performance arguments depending on what
operations you do most commonly.  A theoretically (but probably horrible
in practice) idea might be to consider "DB query latency" vs transfer
speed. The metadata needs to cover 99% of common searches and deliver
results quickly - as such it needs to be "near" Dovecot and cover the
main message headers, and perhaps also message body structure (list of
parts, etc).  Next tier down is probably satisfying full text searches
of message bodies and supplying body parts (where the most common
queries might be either just text/html parts (smart clients) or the
entire body (common clients).  Some kind of compressed blocking format
for message bodies is probably most optimal and storing larger
attachments separately would be an interesting way to increase cache hit
ratios.

I don't know whether Timo is interested in working such a project, but I
for one would be interested to sponsor some work on robust async
replication, perhaps there is some crossover with synchronous
replication that you desire?

I think this is an interesting area to develop.  Cyrus has done some
work on this stuff - I haven't followed it, but would be interesting to
see what they have done?

Good luck

Ed W