Hi
The updated design #2 should address this. The mailbox synchronization step works pretty much the same as QRESYNC.
Thanks will look through that
a) Add a FETCH-SMALL command that drops large attachements and somehow remembers this so that they could later be downloaded again when there's more bandwidth or user requests it.
Agreed - it's tricky.
Not 100% sure myself how to handle this yet. I think it requires some extensions quite a long way outside of the normal IMAP process, but it's probably a very nice feature for those that need it.
Using IMAP protocol for replication has at least two disadvantages:
- It's a bit too chatty, wasting bandwidth on replies the replication isn't interested in.
Compression eliminates a huge amount of traffic (I typically see 12:1 or better). Also pipelining commands eliminates much of the disadvantage of the chatty behaviour (lets assume >1sec latency on most dialup links, so latency is definitely a killer)
I currently use a small self written proxy app which does some simple analysis of what imap client is talking and does some prefetching via pipelined commands to reduce latency and also sets up a compressed pipe back to the server. Even over broadband it gives quite a significant speedup on large folders and on dialup it gives a huge performance boost. Nothing too clever going on though
Perhaps we could look at some optimisations like that in the first instance?
- Sending updated flag/keyword changes can't be done in a standard way, because it only shows the last flag+keyword state, not the changes that were done (e.g. "\Seen" vs. "+\Seen -\Flagged").
Hmm... I guess that can only be done anyway by storing the state before and after and figuring out the changes based on a comparison?
I do like the idea of making this more generic and hence hackable than writing all the code into dovecot itself. Perhaps we could start with an external proxy app at each end of the link which is external to the imap server, ie basically start with IMAP sync. This seems easy to knock up in teh scripting language of choice (eg imapsync) and we can then easily hack on the protocol and choice of commands to bring the servers into sync. I guess if some obvious bottlenecks occur then it's simple to make the protocol across the wire slightly different and subsequently look at how those changes could be moved into the imap protocol itself?
The proxy app then gives a clean break to monitor stuff like changes in flags (expecting this ot need some support from the server though to avoid duplicating the index data?). It would potentially make it possible to support other imap servers than dovecot, although I don't believe that should be on the roadmap, but others may want to code that up themselves?
So effectively start with ImapSync style app and then use knowledge of the IMAP server and the QRESYNC stuff to make it very much more optimised. Other imap servers then have the option to code up the requried missing features and we have invented a standardised way to sync two servers...
Sound any good?
Ed W