Can I make a very weak suggestion to look at that ZLIB compression extension I think you mentioned in the past?
It would have to be done by proxying in imap-login similar to how SSL connections are handled. But aren't you using SSL already, and why not? Using that would give compression for free. Although I haven't really looked at if it's already automatically enabled or if I or clients should do something special..
I don't think that SSL in general has compression enabled? Could be wrong, but I believe it's a option, but badly supported? I'm not an expert though so I don't know that for sure... I would be interested if someone had a recipe for enabling compression on TLS?
Also if you use SSL then you can no longer do after the fact compression. By definition, encryption done well produces an output which cannot be compressed. So it's even more important to precompress before encryption
Anyway, just a thought - I'm assuming that the probable implementation is going to be fairly simple. I would think that zlib and/or lzo would be good compressors if there is a choice of implementations? Certainly LZO would be a good choice for faster 100mbit connections
http://www.ietf.org/rfc/rfc4978.txt specifies DEFLATE format that can be implemented using zlib.
I think this is probably what you referenced before.
My own experience is using a very powerful (cpu hungry) compressor where it doesn't seem to matter all that much if stuff is base64 encoded or not. Long shot is that whilst all that reflushing sounds really nice I think it's just icing compared with just doing blind compression of everything...
My guess is that with the replication stuff you are going to see a 5x-10x speedup on exchanging long lists of guids to compare folders, etc. Compression on the actual mailbodies may be much less. In my case even with an incompressible jpg file which is base64 encoded we still knock off the expected 1/3 in file size due to the base64 encoding so it's a nice benefit
(My customers are on dialup connections of just 2,400 baud... ie 20KB per *minute* http://www.mailasail.com )
For replication I would have thought support of an optional non RFC LZO compressor would be beneficial on anything under gigabit links..?
Ed W