[Dovecot] dsync redesign

[Jan-Frode Myklebust]

On Sat, Mar 24, 2012 at 08:19:48AM +0100, Attila Nagy wrote:
> On 03/23/12 22:25, Timo Sirainen wrote:
> >
> Well, dsync is a very useful tool, but with continuous replication
> it tries to solve a problem which should be handled -at least
> partially- elsewhere. Storing stuff in plain file systems and
> duplicating them to another one just doesn't scale.

I don't see why this shouldn't scale. Mailboxes are after all changed
relatively infrequently. One idea for making it more scalable might be
to treat indexes/metadata and messages differently. Make index/metadata
updates synchronous over the clusters/locations (with re-sync capability
in case of lost synchronisation), while messages are store in one
"altstorage" per cluster/location.

For a two-location solution, message-data should be stored in:

	mail_location = mdbox:~/mdbox
			ALTcache=mdbox:~/mdbox-remoteip-cache
			ALT=dfetch://remoteip/   <-- new protocol

If a message is in the index, look for it in that order:

	local mdbox
	ALTcache
	ALT

if it finds the message in ALT, make a copy into ALTcache (or local
mdbox?).

Syncronizing messages could be a very low frequency job, and could be
handled by simple rsync of ALT to ALTcache. No need for specialized tool
for this job. Syncronizing ALTcache to local mdbox could be done with a
reversed doveadm-altmove, but might not be necessary.

Of course this is probably all very naive.. but you get the idea :-)


   -jf