[Dovecot] Capability COMPRESS implemented?

[Ed W]

Timo Sirainen wrote:
> On Thu, 2009-06-25 at 21:49 +0100, Ed W wrote:
>   
>>>> Timo normally chimes in pretty fast on these types of questions - Any 
>>>> chance of a yay/nay on the COMPRESS option Timo?
>>>>         
>>> Maybe. I'm kind of busy with other stuff though..
>>>
>>>   
>>>       
>> Understood
>>
>> Please take it as a +1 interested here.  I guess you don't take external 
>> paid work now...
>>     
>
> Yeah, not for next half a year at least. Anyway, it would basically need
> istream and ostream implementations for zlib. istream implementation
> kind of already exists in zlib plugin, except it's using gz*() functions
> instead of doing everything in memory. So:
>
>  - create zlib istream using zlib's deflate*() functions (I think?) and
> which takes another istream as input
>  - convert zlib plugin to use that stream instead
>  - implement zlib ostream
>  - create yet another proxy to login processes. Probably some day I
> should combine all of them to one that only proxies i/ostreams. Although
> implementing SSL i/ostreams could be a bit difficult.
>   

OK, so I gave the developers of profimail (a rather neat imap client for 
Nokia symbian phones, decent idle support, etc) a nudge about the recent 
thread here on compression support and they tell me that they have no 
knobs or bells to influence the SSL implementation under symbian, so 
apparently no SSL compression is available on symbian (boo). 

However, I also pointed out the COMPRESS rfc and 6 hours later they sent 
me a new build with COMPRESS support!  It's tested against cyrus and 
fastmail in particular

I promised Timo some notes on zlib implementation many months back - 
this was about to get me off my chair until I noticed there is an 
excellent starting guide on the zlib site:
    http://www.zlib.net/zlib_how.html

So essentially an in-memory compressor is extremely simple:

- Start with deflateInit to get some in memory data structures working 
(pseudo object based library...)
- The datastructure has pointers to an input buffer and an output buffer 
plus separate counters of bytes remaining in each
- Call deflate (or inflate) as many times as you need. The library will 
return result codes so you know whether you run out of input or output 
space.
- Alter/flush your buffers as appropriate and keep calling deflate until 
you are done

- Obviously at some point you have fed in all your input and no more is 
waiting, so you need to flush down the compressor to get any remaining 
bytes out of it.  Call deflate with the flush param (hopefully it's 
clear that you can feed a load of bytes into a compressor and get 
absolutely nothing out of the other end, basically if the algorithm has 
a good prediction going - the flush just says to clear down and assume 
no more to come for the time being - flush as little as you can though)
- Finally at some point we need to close the stream, so call deflateEnd 
to achieve this



Does this help?

It would appear that the whole DEFLATE RFC is really designed just to 
turn on a wholesale compressed tunnel of data once you see the correct 
incantation, so this would appear to belong somewhere in the output 
layer near wherever dovecot writes to it's socket?

Thanks for listening

Ed W