Timo Sirainen wrote:
On Thu, 2009-06-25 at 21:49 +0100, Ed W wrote:
Timo normally chimes in pretty fast on these types of questions - Any chance of a yay/nay on the COMPRESS option Timo?
Maybe. I'm kind of busy with other stuff though..
Understood
Please take it as a +1 interested here. I guess you don't take external paid work now...
Yeah, not for next half a year at least. Anyway, it would basically need istream and ostream implementations for zlib. istream implementation kind of already exists in zlib plugin, except it's using gz*() functions instead of doing everything in memory. So:
- create zlib istream using zlib's deflate*() functions (I think?) and which takes another istream as input
- convert zlib plugin to use that stream instead
- implement zlib ostream
- create yet another proxy to login processes. Probably some day I should combine all of them to one that only proxies i/ostreams. Although implementing SSL i/ostreams could be a bit difficult.
OK, so I gave the developers of profimail (a rather neat imap client for Nokia symbian phones, decent idle support, etc) a nudge about the recent thread here on compression support and they tell me that they have no knobs or bells to influence the SSL implementation under symbian, so apparently no SSL compression is available on symbian (boo).
However, I also pointed out the COMPRESS rfc and 6 hours later they sent me a new build with COMPRESS support! It's tested against cyrus and fastmail in particular
I promised Timo some notes on zlib implementation many months back - this was about to get me off my chair until I noticed there is an excellent starting guide on the zlib site: http://www.zlib.net/zlib_how.html
So essentially an in-memory compressor is extremely simple:
Start with deflateInit to get some in memory data structures working (pseudo object based library...)
The datastructure has pointers to an input buffer and an output buffer plus separate counters of bytes remaining in each
Call deflate (or inflate) as many times as you need. The library will return result codes so you know whether you run out of input or output space.
Alter/flush your buffers as appropriate and keep calling deflate until you are done
Obviously at some point you have fed in all your input and no more is waiting, so you need to flush down the compressor to get any remaining bytes out of it. Call deflate with the flush param (hopefully it's clear that you can feed a load of bytes into a compressor and get absolutely nothing out of the other end, basically if the algorithm has a good prediction going - the flush just says to clear down and assume no more to come for the time being - flush as little as you can though)
Finally at some point we need to close the stream, so call deflateEnd to achieve this
Does this help?
It would appear that the whole DEFLATE RFC is really designed just to turn on a wholesale compressed tunnel of data once you see the correct incantation, so this would appear to belong somewhere in the output layer near wherever dovecot writes to it's socket?
Thanks for listening
Ed W