Timo Sirainen escreveu:

If your OpenSSL supports it, Dovecot supports it. I recently tested this with gnutls-cli program, openssl s_client for some reason didn't support it. I've no idea if any actual IMAP clients support it.

i'm using OpenSSL shipped from CentOS 5.3 ..... is there any easy to 

check if the shipped OpenSSL supports that ???

--

Atenciosamente / Sincerily,
Leonardo Rodrigues
Solutti Tecnologia
http://www.solutti.com.br

Minha armadilha de SPAM, NÃO mandem email
gertrudes@solutti.com.br
My SPAMTRAP, do not email it

Timo Sirainen wrote:

On Sep 22, 2009, at 9:08 PM, Leonardo Rodrigues wrote:

...

is there anything that can be easily used to automatically compress IMAP traffic between client and server ? I was thinking if the SSL/TLS code enables some kind of compression as well.

If your OpenSSL supports it, Dovecot supports it. I recently tested this with gnutls-cli program, openssl s_client for some reason didn't support it. I've no idea if any actual IMAP clients support it.

I think this kind of feature is potentially more interesting than people give it credit. Timo, would it be possible to get some additional debugging on the SSL connection creation so that we can see if compression was enabled? Packet tracing is all well and good, but can be complicated for a non local production server, etc (mine are also virtualised machines which also increases the difficulty of packet tracing correctly)

It's not really clear why openssl isn't supporting this correctly, but I think there is a quite a large performance boost to be had on even decently fast broadband connections if this could suddenly start working across a wide range of clients...

I would be interested to hear if anyone found there a solution to why openssl isn't enabling this compression?

All the best

Ed W

Timo Sirainen wrote:

On Mon, 2009-09-28 at 16:01 +0100, Ed W wrote:

...

...

If your OpenSSL supports it, Dovecot supports it. I recently tested this with gnutls-cli program, openssl s_client for some reason didn't support it. I've no idea if any actual IMAP clients support it.

I think this kind of feature is potentially more interesting than people give it credit. Timo, would it be possible to get some additional debugging on the SSL connection creation so that we can see if compression was enabled?

http://hg.dovecot.org/dovecot-1.2/rev/26ca4ff5d269

login_log_format_elements = user=<%u> method=%m rip=%r lip=%l %c %k

Thankyou for that - I probably can't test this for some days, but I can't thankyou enough for the super fast response from your side!!

Ed W

Ed W escreveu:

I notice that the openssl docs require compression to be specifically enabled and are somewhat scathing about support...

http://www.openssl.org/docs/ssl/SSL_COMP_add_compression_method.html

Anyone care to comment further?

When i created this thread, some weeks ago, i have also made some 

tests with recent versions (OpenSSL, dovecot, Thunderbird) and couldnt get compression.

I didnt any kind of tracing debug or similar, i just created some IP 

accounting rules and watchs it when downloading a know set of emails with a know size. There was no difference when downloading through unsecure IMAP or secured (TLS) IMAP. So, there's no compression being activated.

When searching for that, i found that there's already a RFC for a 

COMPRESS imap extension ... as imagined, there are pretty few clients that supports it .... Thunderbird 3 Beta supports it .... but asking customers to use a Beta software is not acceptable. So, we'll probably need some more years to have this extensions widely deployed and supported by clients.

http://www.ietf.org/rfc/rfc4978.txt

--

Atenciosamente / Sincerily,
Leonardo Rodrigues
Solutti Tecnologia
http://www.solutti.com.br

Minha armadilha de SPAM, NÃO mandem email
gertrudes@solutti.com.br
My SPAMTRAP, do not email it

Hmm, strange results.

My dovecot compiled on freebsd using openssl doesn't do compression. But my dovecot compiled on redhat using openssl does do it.

redhat openssl 0.9.8b freebsd openssl 0.9.7e (really old)

Quoting Ed W <lists@wildgooses.com>:

Timo Sirainen wrote:

...

On Sep 22, 2009, at 9:08 PM, Leonardo Rodrigues wrote:

...

is there anything that can be easily used to automatically
compress IMAP traffic between client and server ? I was thinking
if the SSL/TLS code enables some kind of compression as well.

If your OpenSSL supports it, Dovecot supports it. I recently tested
this with gnutls-cli program, openssl s_client for some reason
didn't support it. I've no idea if any actual IMAP clients support
it.

I notice that the openssl docs require compression to be specifically enabled and are somewhat scathing about support...

http://www.openssl.org/docs/ssl/SSL_COMP_add_compression_method.html

Anyone care to comment further?

Ed

Timo Sirainen escreveu:

I think the compression support in OpenSSL is relatively new, so it's entirely possible that it's only in v0.9.8 and newer.

from a fully upgraded CentOS 5.3 x86_64 box:

[root@correio dovecot]# rpm -qi openssl Name : openssl Relocations: (not relocatable) Version : 0.9.8e Vendor: CentOS Release : 12.el5 Build Date: Fri 04 Sep 2009 09:33:56 AM BRT

i have applied the provided patch, recompiled and installed dovecot 

1.2.5 new binaries. This is what i get from logs:

Sep 28 14:44:43 correio dovecot: imap-login: Login: user=<mail@box.com.br>, method=PLAIN, rip=189.114.xx.x, lip=200.140.yy.yy, TLS, TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)

login_log_format_elements was defined, as documented by Timo, as:

login_log_format_elements = user=<%u> method=%m rip=%r lip=%l %c %k

with clients Thunderbird 2.0.0.23 and Windows Live Mail from a 

Windows Vista SP2 fully updated too, log is the same. There's no trace of compression being enabled.

--

Atenciosamente / Sincerily,
Leonardo Rodrigues
Solutti Tecnologia
http://www.solutti.com.br

Minha armadilha de SPAM, NÃO mandem email
gertrudes@solutti.com.br
My SPAMTRAP, do not email it

Timo Sirainen escreveu:

See if you can get gnutls-cli from somewhere (gnutls-utils package I think?). Using the gnutls-cli command from my previous mail would show if your OpenSSL is at least able to use compression. Anyway I wouldn't be surprised if you couldn't find any clients that are really able to use compression.

i got gnutls-cli from gnutls-utils package ...  but it's probably a 

different version from yours, because yours exactly command line gives error:

[root@correio dovecot]# gnutls-cli --priority NORMAL:+COMP-DEFLATE --insecure -p 993 localhost Invalid option 'priority' Error in the arguments. Use the --help or -h parameters to get more information. [root@correio dovecot]#

[root@correio dovecot]# gnutls-cli --version
GNU TLS test client, version 1.4.1. Libgnutls 1.4.1. [root@correio dovecot]#

from man page, i have the option:

   --comp comp1 comp2...
          Compression methods to enable (use gnutls-cli --list to 

show the supported compression methods).

--list gives

[root@correio dovecot]# gnutls-cli --list

Certificate types: X.509, OPENPGP Protocols: TLS1.0, TLS1.1, SSL3.0 Ciphers: AES-256-CBC, AES-128-CBC, 3DES-CBC, ARCFOUR, ARCFOUR-40 MACs: MD5, RMD160, SHA1 Key exchange algorithms: RSA, RSA-EXPORT, DHE-DSS, DHE-RSA, DHE-PSK, PSK, SRP, SRP-RSA, SRP-DSS, ANON-DH Compression methods: DEFLATE, LZO, NULL [root@correio dovecot]#

trying LZO and DEFLATE gives an error:

[root@correio dovecot]# gnutls-cli --insecure -p 993 localhost --comp LZO
Resolving 'localhost'... Connecting to '127.0.0.1:993'... *** Fatal error: A TLS fatal alert has been received. *** Received alert [50]: Decode error *** Handshake has failed GNUTLS ERROR: A TLS fatal alert has been received. [root@correio dovecot]#

and in maillog:

Sep 28 15:35:05 correio dovecot: imap-login: Disconnected (no auth attempts): rip=127.0.0.1, lip=127.0.0.1, TLS handshaking: SSL_accept() failed: error:1408A0BB:SSL routines:SSL3_GET_CLIENT_HELLO:no compression specified

do the IMAP4 server you tried is remotely accessible so i can try 

from a different machine ? Maybe we're dealing with some client lack of compatibility and not server one ......

--

Atenciosamente / Sincerily,
Leonardo Rodrigues
Solutti Tecnologia
http://www.solutti.com.br

Minha armadilha de SPAM, NÃO mandem email
gertrudes@solutti.com.br
My SPAMTRAP, do not email it

Timo Sirainen escreveu:

And DEFLATE gives the exact same error? LZO isn't supported by OpenSSL.

yes ... error from DEFLATE and LZO are exactly the same on 

gnutls-cli output and maillog on the CentOS 5.3 box.

Well, not the same server but looks like this one works too:

gnutls-cli --priority NORMAL:+COMP-DEFLATE -p 993 secure.emailsrvr.com

And just for fun I tried imap.gmail.com, that didn't support compression.

i had tried imap.gmail.com too :)

interesting findings ..... from CentOS 5.3, i cant get any 

compression method to work:

[root@correio dovecot]# gnutls-cli --insecure -p 993 -p 993 secure.emailsrvr.com --comp LZO DEFLATE NULL [ ......]

• Version: TLS 1.0

• Key Exchange: DHE RSA

• Cipher: AES 256 CBC

• MAC: SHA

• Compression: NULL

  but from a Fedora 8 box:

[root@correio ~]# gnutls-cli --insecure -p 993 -p 993 secure.emailsrvr.com --comp LZO DEFLATE NULL [ ......]

• Version: TLS 1.0

• Key Exchange: DHE RSA

• Cipher: AES 256 CBC

• MAC: SHA

• Compression: DEFLATE

  and Fedora 8 OpenSSL is even older than CentOS 5.3 one:

CentOS 5.3: [root@correio dovecot]# rpm -qi openssl Name : openssl Relocations: (not relocatable) Version : 0.9.8e Vendor: CentOS Release : 12.el5 Build Date: Fri 04 Sep 2009 09:33:56 AM BRT

Fedora 8: [root@correio ~]# rpm -qi openssl Name : openssl Relocations: (not relocatable) Version : 0.9.8b Vendor: Fedora Project Release : 17.fc8 Build Date: Mon 15 Oct 2007 07:56:22 PM BRST

probably there's some build option on CentOS that is disabling 

compression. If 0.9.8b on Fedora8 built in October/2007 can do it, so 0.9.8e on CentOS 5.3 built on September/2009 should be able to do it too ....... oh boy, i really hate those weirds compilation options from Redhat .... :\

--

Atenciosamente / Sincerily,
Leonardo Rodrigues
Solutti Tecnologia
http://www.solutti.com.br

Minha armadilha de SPAM, NÃO mandem email
gertrudes@solutti.com.br
My SPAMTRAP, do not email it

The command I used was:

gnutls-cli --protocols NORMAL:+COMP-DEFLATE --insecure -p 993

I have tried the --comp option, but it always fails for me (ubuntu 8.04)

gnutls-cli (GnuTLS) 2.0.4

Redhat is 5.3 Freebsd is 6.3

Quoting Leonardo Rodrigues <leolistas@solutti.com.br>:

Timo Sirainen escreveu:

...

And DEFLATE gives the exact same error? LZO isn't supported by OpenSSL.

yes ... error from DEFLATE and LZO are exactly the same on gnutls-cli output and maillog on the CentOS 5.3 box.

...

Well, not the same server but looks like this one works too:

gnutls-cli --priority NORMAL:+COMP-DEFLATE -p 993 secure.emailsrvr.com

And just for fun I tried imap.gmail.com, that didn't support compression.

i had tried imap.gmail.com too :)

interesting findings ..... from CentOS 5.3, i cant get any compression method to work:

[root@correio dovecot]# gnutls-cli --insecure -p 993 -p 993 secure.emailsrvr.com --comp LZO DEFLATE NULL [ ......]

• Version: TLS 1.0

• Key Exchange: DHE RSA

• Cipher: AES 256 CBC

• MAC: SHA

• Compression: NULL

  but from a Fedora 8 box:

[root@correio ~]# gnutls-cli --insecure -p 993 -p 993 secure.emailsrvr.com --comp LZO DEFLATE NULL [ ......]

• Version: TLS 1.0

• Key Exchange: DHE RSA

• Cipher: AES 256 CBC

• MAC: SHA

• Compression: DEFLATE

  and Fedora 8 OpenSSL is even older than CentOS 5.3 one:

CentOS 5.3: [root@correio dovecot]# rpm -qi openssl Name : openssl Relocations: (not relocatable) Version : 0.9.8e Vendor: CentOS Release : 12.el5 Build Date: Fri 04 Sep 2009 09:33:56 AM BRT

Fedora 8: [root@correio ~]# rpm -qi openssl Name : openssl Relocations: (not relocatable) Version : 0.9.8b Vendor: Fedora Project Release : 17.fc8 Build Date: Mon 15 Oct 2007 07:56:22 PM BRST

probably there's some build option on CentOS that is disabling compression. If 0.9.8b on Fedora8 built in October/2007 can do it, so 0.9.8e on CentOS 5.3 built on September/2009 should be able to do it too ....... oh boy, i really hate those weirds compilation options from Redhat .... :\

--

Atenciosamente / Sincerily, Leonardo Rodrigues Solutti Tecnologia http://www.solutti.com.br

Minha armadilha de SPAM, NÃO mandem email gertrudes@solutti.com.br My SPAMTRAP, do not email it

Ok last info.

using OpenSSL 0.9.8g openssl s_client -connect host:993

New, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-SHA Server public key is 2048 bit Compression: zlib compression Expansion: zlib compression SSL-Session: Protocol : SSLv3 Cipher : DHE-RSA-AES256-SHA Session-ID:
1E5412EC32463E66FC75D761A4D48CF6ED416187F32A81F6DAC3DA4E9028E2DE Session-ID-ctx: Master-Key:
B0E15199867D8B48F31F8776C7E439542F4D1A7B33239814CE0C5FF564CB007DE431E9357DF120E7AF347CD1E934CE83 Key-Arg : None Compression: 1 (zlib compression) Start Time: 1254198546 Timeout : 7200 (sec)

Quoting Patrick Domack <patrickdk@patrickdk.com>:

Just playing some more and noticed using: gnutls-cli (GnuTLS) 2.4.2

always says compression isn't supported, even when version 2.0.4 says it was.

gnutls-cli 2.4.2 from ubuntu 9.04 x64, Compression: DEFLATE, NULL

gnutls-cli 2.0.4 from ubuntu 8.04 x64, Compression: LZO, DEFLATE, NULL

I also noticed 2.4.2 would connect using aes-128, whereas 2.0.4 would connect using aes-256

Quoting Patrick Domack <patrickdk@patrickdk.com>:

...

The command I used was:

gnutls-cli --protocols NORMAL:+COMP-DEFLATE --insecure -p 993

I have tried the --comp option, but it always fails for me (ubuntu 8.04)

gnutls-cli (GnuTLS) 2.0.4

Redhat is 5.3 Freebsd is 6.3

Quoting Leonardo Rodrigues <leolistas@solutti.com.br>:

...

Timo Sirainen escreveu:

...

And DEFLATE gives the exact same error? LZO isn't supported by OpenSSL.

yes ... error from DEFLATE and LZO are exactly the same on gnutls-cli output and maillog on the CentOS 5.3 box.

...

Well, not the same server but looks like this one works too:

gnutls-cli --priority NORMAL:+COMP-DEFLATE -p 993 secure.emailsrvr.com

And just for fun I tried imap.gmail.com, that didn't support compression.

i had tried imap.gmail.com too :)

interesting findings ..... from CentOS 5.3, i cant get any compression method to work:

[root@correio dovecot]# gnutls-cli --insecure -p 993 -p 993 secure.emailsrvr.com --comp LZO DEFLATE NULL [ ......]

• Version: TLS 1.0
• Key Exchange: DHE RSA
• Cipher: AES 256 CBC
• MAC: SHA
• Compression: NULL

but from a Fedora 8 box:

[root@correio ~]# gnutls-cli --insecure -p 993 -p 993 secure.emailsrvr.com --comp LZO DEFLATE NULL [ ......]

• Version: TLS 1.0
• Key Exchange: DHE RSA
• Cipher: AES 256 CBC
• MAC: SHA
• Compression: DEFLATE

and Fedora 8 OpenSSL is even older than CentOS 5.3 one:

CentOS 5.3: [root@correio dovecot]# rpm -qi openssl Name : openssl Relocations: (not relocatable) Version : 0.9.8e Vendor: CentOS Release : 12.el5 Build Date: Fri 04 Sep 2009 09:33:56 AM BRT

Fedora 8: [root@correio ~]# rpm -qi openssl Name : openssl Relocations: (not relocatable) Version : 0.9.8b Vendor: Fedora Project Release : 17.fc8 Build Date: Mon 15 Oct 2007 07:56:22 PM BRST

probably there's some build option on CentOS that is disabling compression. If 0.9.8b on Fedora8 built in October/2007 can do it, so 0.9.8e on CentOS 5.3 built on September/2009 should be able to do it too ....... oh boy, i really hate those weirds compilation options from Redhat .... :\

--

Atenciosamente / Sincerily, Leonardo Rodrigues Solutti Tecnologia http://www.solutti.com.br

Minha armadilha de SPAM, NÃO mandem email gertrudes@solutti.com.br My SPAMTRAP, do not email it

well ..... here for me, with 'openssl s_client', i cant even connect 

when using -ssl2:

[root@correio ~]# openssl s_client -connect localhost:993 -ssl2 [ ... ] 27110:error:1406D0B8:SSL routines:GET_SERVER_HELLO:no cipher list:s2_clnt.c:450: [root@correio ~]#

but that's probably because i have on dovecot.conf:

ssl_cipher_list = ALL:!LOW:!SSLv2

with ssl3 and tls1 i can connect and see the zlib compression being 

enabled.

SSL-Session: Protocol : SSLv3 Cipher : DHE-RSA-AES256-SHA [ ..... ] Compression: 1 (zlib compression)

SSL-Session: Protocol : TLSv1 Cipher : DHE-RSA-AES256-SHA [ ..... ] Compression: 1 (zlib compression)

Thunderbird has the options to enable/disable each cipher of 

ssl2/ssl3/tls1 as well as disable them completly too. Here in my Thunderbird 2.0.0.23, SSLv2 is disabled, and this is certainly the default configs, as i never tweaked this.

http://img43.imageshack.us/img43/7937/thunderbirdssl2.jpg

logging from dovecot shows clearly that i'm using TLSv1 to connect 

... it also shows that TLSv1 connections from thunderbird do not use compression, and connections from gnutls-cli correctly enables that

thunderbird 2.0.0.23 Sep 29 07:12:02 correio dovecot: imap-login: Login: user=<mail@box.com.br>, method=PLAIN, rip=189.114.xx.xx, lip=200.140.xx.xx, TLS, TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)

gnutls-cli Sep 28 18:36:54 correio dovecot: imap-login: Login: user=<mail@box.com.br>, method=PLAIN, rip=189.11.xx.xx, lip=200.140.xx.xx, TLS, TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits) zlib compression

wireshack confirms i'm using TLSv1 and also shows Thunderbird is 

announcing no compression is supported.

http://img33.imageshack.us/img33/9011/wiresharktlsv1.jpg

so ..... despite the known fact that SSLv2 cant be used if 

compression is wanted, using SSLv3 and TLSv1 apparently does not automatically guarantees that .....

Patrick Domack escreveu:

More testing, seems all my imap clients attempt to use ssl2 first, and from the openssl mailing list:

Oops, should've made this clearer. It is only clients than need to avoid the old SSLv2 compatible methods and only use SSLv3/TLSv1. Nothing needs to be done to a server. http://www.mail-archive.com/openssl-users@openssl.org/msg49926.html

This is confirmed using openssl s_client -connect host:993 (-ssl3|-tls1|-ssl2)

I don't see any way around this globally, unless each program has a config option to disable ssl2.

--

Atenciosamente / Sincerily,
Leonardo Rodrigues
Solutti Tecnologia
http://www.solutti.com.br

Minha armadilha de SPAM, NÃO mandem email
gertrudes@solutti.com.br
My SPAMTRAP, do not email it

On Sep 29, 2009, at 6:57 AM, Timo Sirainen wrote:

So I guess if client is using SSLv23_client_method() instead of
SSLv3_client_method() or TLSv1_client_method() it doesn't work. Also
Thunderbird uses Network Security Services library instead of
OpenSSL, so it might not support compression at all.

Wonder if attached patch would enable compression on any client? It's
probably not really the right way to do it, because it breaks at least
openssl s_client unless -tls1 or -ssl3 or -no_ssl2 is given. Looks
like UW-IMAP uses SSLv23_server_method() on imaps/pop3s port, while
TLSv1_server_method() is used for STARTTLS. I think I'll do the same
in Dovecot v2.0 in an case.

2009/9/28 Patrick Domack <patrickdk@patrickdk.com>

Hmm, strange results.

My dovecot compiled on freebsd using openssl doesn't do compression. But my dovecot compiled on redhat using openssl does do it.

redhat openssl 0.9.8b freebsd openssl 0.9.7e (really old)

You don't say which version of FreeBSD you using but ports has openssl v0.9.8k and my old-ish v7.1 box comes with openssl v0.9.8e.

.warren