[Dovecot] Replication protocol design

[Timo Sirainen]

On Mon, 2008-04-28 at 16:33 -0700, Asheesh Laroia wrote:
> Multi-master will be very interesting.  I'm really curious what you will 
> do as far as this scenario:
> 
> 1. A mail arrives in server1
> 2. User reads it (therefore the mail has a UID assigned)

I also thought about this kind of a late UID allocation (or late UID
conflict resolution), but it probably becomes too complex and it should
happen only rarely anyway. So the UID will be assigned immediately when
mail is being stored.

> 3. server2 gets isolated from server1
> 4. A mail arrives at server2
> 5. User logs in to server2 and reads it (therefore the mail has a UID 
> assigned)
> 6. server2 and server1 are connected again, and are told to sync
> 
> Questions:
> 
> a. Does the client get told two different messages have the same UID in 
> the folder?

If user is able to log into either server1 or server2 then there's
really nothing that can be done to avoid seeing the same UID contain
different messages. But I think in normal situations if a user is able
to connect to either one of the servers, the servers should be able to
connect to each others as well.

In step 6 the UID conflict will be noticed and all conflicting messages
be given new unused UIDs so that caching clients won't be confused.

> The above scenario may be handled by your root-based election process - 
> but what prevents two roots from simultaneously existing, and therefore 
> two masters existing, and then the above scenario happening?

The master process tries to prevent the situation from happening in
normal conditions to avoid conflict resolution, but it doesn't rely on
it working.

> I'm really interested to see where this goes - I think this would be 
> really swell as a replacement for my use of offlineimap.  I also want to 
> have you consider letting non-root users use Dovecot replication features; 
> that way, I could just have a cron job that asks my laptop's Dovecot to 
> replicate from my server's Dovecot, without giving it any special 
> permissions.

Hmm. The replication itself could probably be done pretty safely. Mainly
by just not allowing user to become replication master. But it would
have to be treated somewhat differently from normal replication servers,
like not writing (long) replication logs to disk while waiting for it to
become online.

I also thought about using IMAP protocol for initiating the replication
and performing the replication using some IMAP extensions. But it might
be a bit too chatty/bloaty. But maybe for user-initiated replication it
would be useful. Regular IMAP login and after a X-REPLICATE command it
executes the replication binary. Or maybe just make the replication
server directly listen on a different port using a different protocol.
I'm not sure yet. :)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url : http://dovecot.org/pipermail/dovecot/attachments/20080429/820ca52d/attachment.bin