[Dovecot] Capability COMPRESS implemented?
Hi, I just read that http://tools.ietf.org/html/rfc4978
..is being implemented in Thunderbird 3. This is great news!
It's been a long time since we last discussed this and I don't think it's currently implemented in Dovecot, but since Cyrus just added it, any chance of getting support in Dovecot please? (Fastmail seems to be supporting this effort)
On a similar note we had a previous thread about supporting TLS deflate
- any chance you could investigate this again and add support on the Dovecot side so at least it would be feasible to submit a bug fix on the Thunderbird side? (It looked like a one liner change back when we last discussed it?)
Thanks
Ed W
(Obviously for those looking for an external high performance compressor you can use our compressing email proxy from http://www.mailasail.com ...)
Ed W wrote:
Hi, I just read that http://tools.ietf.org/html/rfc4978
..is being implemented in Thunderbird 3. This is great news!
Oh, also looks like CONDSTORE made it to TB3
Wohoo for open standards! (Thanks Timo)
Ed W
On 24.06.2009 2:00, Ed W wrote:
Hi, I just read that http://tools.ietf.org/html/rfc4978
..is being implemented in Thunderbird 3. This is great news!
It's been a long time since we last discussed this and I don't think it's currently implemented in Dovecot, but since Cyrus just added it, any chance of getting support in Dovecot please? (Fastmail seems to be supporting this effort)
On a similar note we had a previous thread about supporting TLS deflate
- any chance you could investigate this again and add support on the Dovecot side so at least it would be feasible to submit a bug fix on the Thunderbird side? (It looked like a one liner change back when we last discussed it?)
Thanks
Ed W
(Obviously for those looking for an external high performance compressor you can use our compressing email proxy from http://www.mailasail.com ...)
To make it easy for people search which standards TB is supported I've keep updated wiki page. https://wiki.mozilla.org/MailNews:Supported_IMAP_extensions
Nikolay Shopik wrote:
To make it easy for people search which standards TB is supported I've keep updated wiki page. https://wiki.mozilla.org/MailNews:Supported_IMAP_extensions
This is very interesting - thanks! Some really good stuff on the cusp of making it into TB. I really like the look of XLIST - this solves a minor issue which has come up a few times on the list, mainly with naming of Sent vs Sent Items folders.
Timo normally chimes in pretty fast on these types of questions - Any chance of a yay/nay on the COMPRESS option Timo?
Cheers
Ed W
On 25.06.2009 23:16, Ed W wrote:
I really like the look of XLIST - this solves a minor issue which has come up a few times on the list, mainly with naming of Sent vs Sent Items folders.
This is something what should be in IMAP since 90s when it was growing up. Still XLIST not even RFC yet, but already solves probably one of biggest downside of IMAP.
Timo Sirainen wrote:
On Thu, 2009-06-25 at 20:16 +0100, Ed W wrote:
Timo normally chimes in pretty fast on these types of questions - Any chance of a yay/nay on the COMPRESS option Timo?
Maybe. I'm kind of busy with other stuff though..
Understood
Please take it as a +1 interested here. I guess you don't take external paid work now...
Cheers
Ed W
On Thu, 2009-06-25 at 21:49 +0100, Ed W wrote:
Timo normally chimes in pretty fast on these types of questions - Any chance of a yay/nay on the COMPRESS option Timo?
Maybe. I'm kind of busy with other stuff though..
Understood
Please take it as a +1 interested here. I guess you don't take external paid work now...
Yeah, not for next half a year at least. Anyway, it would basically need istream and ostream implementations for zlib. istream implementation kind of already exists in zlib plugin, except it's using gz*() functions instead of doing everything in memory. So:
- create zlib istream using zlib's deflate*() functions (I think?) and which takes another istream as input
- convert zlib plugin to use that stream instead
- implement zlib ostream
- create yet another proxy to login processes. Probably some day I should combine all of them to one that only proxies i/ostreams. Although implementing SSL i/ostreams could be a bit difficult.
Timo Sirainen wrote:
On Thu, 2009-06-25 at 21:49 +0100, Ed W wrote:
Timo normally chimes in pretty fast on these types of questions - Any chance of a yay/nay on the COMPRESS option Timo?
Maybe. I'm kind of busy with other stuff though..
Understood
Please take it as a +1 interested here. I guess you don't take external paid work now...
Yeah, not for next half a year at least. Anyway, it would basically need istream and ostream implementations for zlib. istream implementation kind of already exists in zlib plugin, except it's using gz*() functions instead of doing everything in memory. So:
I might have missed the subtleties since it's a while since I wrote anything against the gz interface, but there shouldn't be much difference between interfaces I think?
The only difference is where the buffering is going surely?
The naive implementation would flush whenever you would normally flush the net buffers, but the notes in the RFC point out that you can in fact shoot for some clever stuff and vary your compression params according to the type of data. Having tried all this stuff a bunch I can say that it's a good thought, but unless you are hyper bandwidth constrained then zlib uses such small buffers that it's really not likely to make more than a tiny difference... (and if you are mega bandwidth constrained then don't use IMAP at all...)
Hope you will put it on your TODO anyway... (pretty please...)
FWIW I notice a significant speedup using our compressing proxy over even a 10mbit connection, so I am pretty sure this will lead to a significant improvement in response speeds for a lot of folks
Ed W
On Thu, 2009-06-25 at 23:21 +0100, Ed W wrote:
Yeah, not for next half a year at least. Anyway, it would basically need istream and ostream implementations for zlib. istream implementation kind of already exists in zlib plugin, except it's using gz*() functions instead of doing everything in memory. So:
I might have missed the subtleties since it's a while since I wrote anything against the gz interface, but there shouldn't be much difference between interfaces I think?
I don't know. I've never written anything using the deflate/inflate*() interfaces. I just quickly looked up from zlib.h that those are probably what's needed.
Timo Sirainen wrote:
On Thu, 2009-06-25 at 23:21 +0100, Ed W wrote:
Yeah, not for next half a year at least. Anyway, it would basically need istream and ostream implementations for zlib. istream implementation kind of already exists in zlib plugin, except it's using gz*() functions instead of doing everything in memory. So:
I might have missed the subtleties since it's a while since I wrote anything against the gz interface, but there shouldn't be much difference between interfaces I think?
I don't know. I've never written anything using the deflate/inflate*() interfaces. I just quickly looked up from zlib.h that those are probably what's needed.
I think what you see as a "stream" is just the API name for a memory buffer. The input output variables point to a struct which is something like:
char *buffer_ptr; long bytes_left_in_buffer;
As you call the function it consumes bytes from the input buffer and may optionally squirt some data into the output buffer. The structs you pass are updated to show the new values. The compress/decompress functions return a value which shows if it's finished doing it's thing or required more output buffer space, etc
I suppose the only subtlety is that the compressor (and decompressor) may keep some bytes in it's internal state (ie unflushed). So if you ask it to compress the string "dovecot" and uncompress the ouput bytes you might only get "dove" (say). The key thing is to call the flush function where it's necessary. However, the unflushed characters are those the compressor thinks it can batch with later input, so clearly you minimise the amount of flushing when dealing with small input strings. In terms of big picture compression though it's a very small decrease in efficiency, but clearly it's desirable to minimise flushes where possible (ie only at the end of each command output would be the obvious solution)
I don't know the internals of dovecot too well, but I would have thought that you would add this the network output abstraction. So you presumably already buffer and spool command output to the network socket, now you simply run the output through gzip before each write and after each read. Note there is some potential efficiency gains in compressing attachments slightly differently to other data, hence the compressor might potentially gain by being nearer the code which is generating network output (the decompressor on input data can clearly be right in the network input code) but my opinion is that this is barely relevant for real users with sensible size emails (the zlib dictionary sizes are just too small to get massive compression ratios)
Hopefully this is a fairly easy thing to insert into the current code path?
Cheers
Ed W
Timo Sirainen wrote:
On Thu, 2009-06-25 at 21:49 +0100, Ed W wrote:
Timo normally chimes in pretty fast on these types of questions - Any chance of a yay/nay on the COMPRESS option Timo?
Maybe. I'm kind of busy with other stuff though..
Understood
Please take it as a +1 interested here. I guess you don't take external paid work now...
Yeah, not for next half a year at least. Anyway, it would basically need istream and ostream implementations for zlib. istream implementation kind of already exists in zlib plugin, except it's using gz*() functions instead of doing everything in memory. So:
- create zlib istream using zlib's deflate*() functions (I think?) and which takes another istream as input
- convert zlib plugin to use that stream instead
- implement zlib ostream
- create yet another proxy to login processes. Probably some day I should combine all of them to one that only proxies i/ostreams. Although implementing SSL i/ostreams could be a bit difficult.
OK, so I gave the developers of profimail (a rather neat imap client for Nokia symbian phones, decent idle support, etc) a nudge about the recent thread here on compression support and they tell me that they have no knobs or bells to influence the SSL implementation under symbian, so apparently no SSL compression is available on symbian (boo).
However, I also pointed out the COMPRESS rfc and 6 hours later they sent me a new build with COMPRESS support! It's tested against cyrus and fastmail in particular
I promised Timo some notes on zlib implementation many months back - this was about to get me off my chair until I noticed there is an excellent starting guide on the zlib site: http://www.zlib.net/zlib_how.html
So essentially an in-memory compressor is extremely simple:
Start with deflateInit to get some in memory data structures working (pseudo object based library...)
The datastructure has pointers to an input buffer and an output buffer plus separate counters of bytes remaining in each
Call deflate (or inflate) as many times as you need. The library will return result codes so you know whether you run out of input or output space.
Alter/flush your buffers as appropriate and keep calling deflate until you are done
Obviously at some point you have fed in all your input and no more is waiting, so you need to flush down the compressor to get any remaining bytes out of it. Call deflate with the flush param (hopefully it's clear that you can feed a load of bytes into a compressor and get absolutely nothing out of the other end, basically if the algorithm has a good prediction going - the flush just says to clear down and assume no more to come for the time being - flush as little as you can though)
Finally at some point we need to close the stream, so call deflateEnd to achieve this
Does this help?
It would appear that the whole DEFLATE RFC is really designed just to turn on a wholesale compressed tunnel of data once you see the correct incantation, so this would appear to belong somewhere in the output layer near wherever dovecot writes to it's socket?
Thanks for listening
Ed W
On Tue, 2009-09-29 at 16:40 +0100, Ed W wrote:
Yeah, not for next half a year at least. Anyway, it would basically need istream and ostream implementations for zlib. istream implementation kind of already exists in zlib plugin, except it's using gz*() functions instead of doing everything in memory. So:
- create zlib istream using zlib's deflate*() functions (I think?) and which takes another istream as input
- convert zlib plugin to use that stream instead
The above is now done. Mainly because seeking backwards in bzip2 compressed mails was broken.
- implement zlib ostream
Done also.
- create yet another proxy to login processes. Probably some day I should combine all of them to one that only proxies i/ostreams. Although implementing SSL i/ostreams could be a bit difficult.
This isn't done.. I've a separate http://hg.dovecot.org/dovecot-2.0-sslstream/ that does ssl proxying with ssl iostreams. It's probably easier to change to support compression proxying as well. Lets see if I can manage to do that now.. Anyway that code would be merged only for v2.1, since it changes things so much.
On Sat, 2010-02-13 at 01:19 +0200, Timo Sirainen wrote:
- create yet another proxy to login processes. Probably some day I should combine all of them to one that only proxies i/ostreams. Although implementing SSL i/ostreams could be a bit difficult.
Looks like COMPRESS=DEFLATE is valid only after login, which made it much easier to implement. It's now in v2.0 hg. Tested that it seems to work with Thunderbird 3.0.1.
http://hg.dovecot.org/dovecot-2.0/rev/29f5567e0a9a
protocol imap { mail_plugins = zlib imap_zlib } plugin { imap_zlib_compress_level = 6 #default }
Only COMPRESS=DEFLATE is advertised, but it actually supports all compression methods supported by zlib plugin. So basically deflate and bz2 (and gz, but it's basically same as deflate).
Hi Timo
Looks like COMPRESS=DEFLATE is valid only after login, which made it much easier to implement. It's now in v2.0 hg. Tested that it seems to work with Thunderbird 3.0.1.
http://hg.dovecot.org/dovecot-2.0/rev/29f5567e0a9a
protocol imap { mail_plugins = zlib imap_zlib }
Oh!! Dur. At long, long last I finally understood what you were telling me when you said you wanted to stop the zlib plugin being file based and move to an iostream! This has been trying to get higher up my todo list to have a closer look at for some time, but I completely misunderstood where/how you had it in mind to implement the compression
- your solution seems very neat and clever!
Thanks for implementing this - this is very exciting! I'm tied up with deadlines on other projects, but extremely keen to have a performance test as soon as possible - would be very interested to hear results from anyone else on the list who gives this a go - as I say our compression proxy gives quite a noticable speedup on larger mailboxes over a 10Mbit connection here and it *should* give quite a zing to Profimail for symbian phones (who kindly specifically added support for this on request)
Cheers!
Ed W
participants (3)
-
Ed W
-
Nikolay Shopik
-
Timo Sirainen