Hi,
I tracked down a tricky bug in dovecot that can cause the imap-login and pop3-login processes to crash on handshake failures. This can be tested by disabling SSLv3 in the dovecot config (ssl_protocols = !SSLv2 !SSLv3) and trying to connect with openssl and forced sslv3 (openssl s_client -ssl3 -connect localhost:995). This would cause a crash.
What was going on is this: In ssl-proxy-openssl.c in line 545 in the function ssl_step() the function ssl_handshake() is called. There SSL_accept() is called. If SSL_accept failes - because a client sent an invalid packet or something the server doesn't support or any other reason - ssl_handle_error() will be called.
ssl_handle_error() will call ssl_proxy_destroy(). ssl_proxy_destroy() will then call ssl_proxy_flush(). And ssl_proxy_flush will call ssl_step() again. Here we have a loop. Now when SSL_accept() gets called again on the same context this is an invalid state for OpenSSL and it crashes.
What to do? In essence, if ssl_proxy_destroy is called it shouldn't try to finish the handshake if the handshake hasn't even started due to an error. This can be done by a simple if check, see attached patch. I think this should do it.
I have seen that a bug that is probably rootet in this has been posted here before regarding ssl3-disabled configs: http://dovecot.org/pipermail/dovecot/2015-March/100188.html
cu,
Hanno Böck http://hboeck.de/
mail/jabber: hanno@hboeck.de GPG: BBB51E42