[Dovecot] Weird Dovecot 1.1.6 + pop3s certificate issues

[Bernhard Schmidt]

Hello everyone,

I've been asked by a colleague to have a look at some extremely weird
dovecot SSL issue they are seeing on one of the student mailservers.

They are running dovecot 1.1.6 (yes, I know, a bit old ...) on SLES 10.2
x86_64 with imap(+starttls), imaps, pop3(+starttls) and pop3s enabled.
Every couple of weeks the pop3s and pop3+starttls part bail out out.
Clients can't connect, when you try openssl s_client you get this:

15960:error:0407006A:rsa routines:RSA_padding_check_PKCS1_type_1:block
type is not 01:rsa_pk1.c:100:
15960:error:04067072:rsa routines:RSA_EAY_PUBLIC_DECRYPT:padding check
failed:rsa_eay.c:699:
15960:error:1408D07B:SSL routines:SSL3_GET_KEY_EXCHANGE:bad
signature:s3_clnt.c:1414:

server logs don't show much except for various SSL_accept() errors,
including

TLS handshaking: SSL_accept() failed: error:1408F455:SSL
routines:SSL3_GET_RECORD:decryption failed or bad record mac
TLS handshaking: SSL_accept() syscall failed: Connection reset by peer
TLS handshaking: SSL_accept() failed: error:1409441B:SSL
routines:SSL3_READ_BYTES:tlsv1 alert decrypt error

imaps and imap+starttls work fine during the whole time (and always
have), although they are much more used.

Restarting dovecot fixes the issue for a couple of weeks. Does anyone
have an idea where to start debugging? As far as I understand the whole
SSL_accept() thing is a giant blackbox for Dovecot, but how comes it
only affects POP3? dovecot -n (paths and addresses altered, not sure how
much I can reveal without getting shot):

# 1.1.6: /mnt//usr/local/etc/dovecot.conf
# OS: Linux 2.6.16.60-0.25-smp x86_64 SUSE Linux Enterprise Server 10
# (x86_64)
protocols: imap imaps pop3 pop3s
listen(default): [ipv6]:143, ipv4:143
listen(imap): [ipv6]:143, ipv4:143
listen(pop3): [ipv6]:110, ipv4:110
ssl_listen(default): [ipv6]:993, ipv4:993
ssl_listen(imap): [ipv6]:993, ipv4:993
ssl_listen(pop3): [ipv6]:995, ipv4:995
disable_plaintext_auth: no
login_dir: /usr/local/var/run/dovecot/login
login_executable(default):
/mnt//usr/local/libexec/dovecot/imap-login
login_executable(imap):
/mnt//usr/local/libexec/dovecot/imap-login
login_executable(pop3):
/mnt//usr/local/libexec/dovecot/pop3-login
login_process_per_connection: no
login_max_connections: 128
max_mail_processes: 2500
mail_uid: campus
mail_gid: lmu
mail_location: maildir:~/Maildir:INDEX=/home/something/indexes/%-1.1n/%n
mmap_disable: yes
mail_nfs_index: yes
mail_executable(default): /mnt//usr/local/libexec/dovecot/imap
mail_executable(imap): /mnt//usr/local/libexec/dovecot/imap
mail_executable(pop3): /mnt//usr/local/libexec/dovecot/pop3
mail_plugins(default): quota imap_quota
mail_plugins(imap): quota imap_quota
mail_plugins(pop3): quota
mail_plugin_dir(default): /mnt//usr/local/lib/dovecot/imap
mail_plugin_dir(imap): /mnt//usr/local/lib/dovecot/imap
mail_plugin_dir(pop3): /mnt//usr/local/lib/dovecot/pop3
pop3_uidl_format(default): %08Xu%08Xv
pop3_uidl_format(imap): %08Xu%08Xv
pop3_uidl_format(pop3): %08Xv%08Xu
namespace:
  type: private
  separator: .
  prefix: INBOX.
  inbox: yes
  list: yes
  subscriptions: yes
auth default:
  worker_max_request_count: 500
  passdb:
    driver: ldap
    args: /mnt//usr/local/etc/dovecot-ldap.conf
  userdb:
    driver: prefetch
  userdb:
    driver: ldap
    args: /mnt//usr/local/etc/dovecot-ldap.conf
  socket:
    type: listen
    master:
      path: /var/run/dovecot/auth-master
      mode: 384
      user: campus
      group: lmu
plugin:
  quota: maildir
  quota_rule: Trash:ignore
  quota_rule2: *:storage=512M

Anyone having an idea?

bernhard