Hi,
After many hours of testing, I've finally tracked down the issue I have been having with dovecot's SSL support. The problem is that the SSL certs result in "TLS handshaking: SSL_accept() syscall failed: Connection reset by peer" errors *if the certificate granted is not granted for client use*.
For servers, I normally generate SSL certificates specifically for servers:
[ server_ca_extensions ] basicConstraints = CA:false keyUsage = keyEncipherment extendedKeyUsage = 1.3.6.1.5.5.7.3.1
If you just do that, then the SSL certificate doesn't work in dovecot (it will work fine in Apache, or Postfix etc etc). You also need the certificate to be valide for client side work:
[ client_and_server_ca_extensions ] basicConstraints = CA:false keyUsage = digitalSignature, keyEncipherment extendedKeyUsage = 1.3.6.1.5.5.7.3.2, 1.3.6.1.5.5.7.3.1
I believe this is a fault with Dovecot. This is with the Debian package for Dovecot 1.2.11 (Version: 1:1.2.11-1). The default self-signed certificate the Debian post-install script generates pops out a certificate that is valid for everything, which is why it works by default out of the box.
Matthew