[Dovecot] Replication protocol design

[Ed W]

Hi

> The updated design #2 should address this. The mailbox synchronization
> step works pretty much the same as QRESYNC.
>   

Thanks will look through that

> a) Add a FETCH-SMALL command that drops large attachements and somehow
> remembers this so that they could later be downloaded again when there's
> more bandwidth or user requests it.
>   

Agreed - it's tricky.

Not 100% sure myself how to handle this yet.  I think it requires some 
extensions quite a long way outside of the normal IMAP process, but it's 
probably a very nice feature for those that need it.

> Using IMAP protocol for replication has at least two disadvantages:
>
> 1) It's a bit too chatty, wasting bandwidth on replies the replication
> isn't interested in.
>   

Compression eliminates a huge amount of traffic (I typically see 12:1 or 
better).  Also pipelining commands eliminates much of the disadvantage 
of the chatty behaviour (lets assume >1sec latency on most dialup links, 
so latency is definitely a killer)

I currently use a small self written proxy app which does some simple 
analysis of what imap client is talking and does some prefetching via 
pipelined commands to reduce latency and also sets up a compressed pipe 
back to the server.  Even over broadband it gives quite a significant 
speedup on large folders and on dialup it gives a huge performance 
boost.  Nothing too clever going on though

Perhaps we could look at some optimisations like that in the first instance?

> 2) Sending updated flag/keyword changes can't be done in a standard way,
> because it only shows the last flag+keyword state, not the changes that
> were done (e.g. "\Seen" vs. "+\Seen -\Flagged").
>   

Hmm... I guess that can only be done anyway by storing the state before 
and after and figuring out the changes based on a comparison?


I do like the idea of making this more generic and hence hackable than 
writing all the code into dovecot itself.  Perhaps we could start with 
an external proxy app at each end of the link which is external to the 
imap server, ie basically start with IMAP sync.  This seems easy to 
knock up in teh scripting language of choice (eg imapsync) and we can 
then easily hack on the protocol and choice of commands to bring the 
servers into sync.  I guess if some obvious bottlenecks occur then it's 
simple to make the protocol across the wire slightly different and 
subsequently look at how those changes could be moved into the imap 
protocol itself? 

The proxy app then gives a clean break to monitor stuff like changes in 
flags (expecting this ot need some support from the server though to 
avoid duplicating the index data?).  It would potentially make it 
possible to support other imap servers than dovecot, although I don't 
believe that should be on the roadmap, but others may want to code that 
up themselves?

So effectively start with ImapSync style app and then use knowledge of 
the IMAP server and the QRESYNC stuff to make it very much more 
optimised.  Other imap servers then have the option to code up the 
requried missing features and we have invented a standardised way to 
sync two servers...

Sound any good?

Ed W