multiple SSL certificates story

[Arkadiusz Miśkiewicz]

Hi.

Little story :-)

I'm playing with dovecot 2.2.25 and multiple SSL certificates. ~7000 certificates
which are loaded twice, so my dovecot has ~14 000 certificate pairs
(14k key + 14k cert) in config.

14 000 local_name entries. Like these:

local_name imap.example.com {
ssl_cert = </etc/certs/cert1.pem
ssl_key = </etc/certs/cert1.pem
}

local_name pop3.example.com {
ssl_cert = </etc/certs/cert1.pem
ssl_key = </etc/certs/cert1.pem
}

(it needs to be "duplicated" like this because dovecot
doesn't support globs/regexps for local_name, so can't use:
    local_name *.example.com {
        ssl_cert = </etc/certs/cert1.pem
        ssl_key = </etc/certs/cert1.pem
    }
)


All certificates take 87MB on disk. Dovecot config for certificates itself is 3MB.

Now fun begins. Trying to start dovecot fails

Oct 26 12:49:59 ixion dovecot[19831]: master: Dovecot v2.2.25 (7be1766) starting up for imap, pop3, lmtp (core dumps disabled)
Oct 26 12:50:01 ixion dovecot[19835]: config: Fatal: pool_system_malloc(1608): Out of memory
Oct 26 12:50:01 ixion dovecot[19831]: master: Error: service(config): command startup failed, throttling for 2 secs
Oct 26 12:50:01 ixion dovecot[19835]: config: Fatal: master: service(config): child 19837 returned error 83 (Out of memory (service config { vsz_limit=1024 MB }, you may need to increase it) - set CORE_OUTOFMEM=1 environment to 
get core dump)
Oct 26 12:50:01 ixion dovecot[19835]: anvil: Fatal: Error reading configuration: read(/var/run/dovecot/config) failed: read(size=8192) failed: Connection reset by peer
Oct 26 12:50:01 ixion dovecot[19835]: ssl-params: Fatal: Error reading configuration: read(/var/run/dovecot/config) failed: read(size=8192) failed: Connection reset by peer
Oct 26 12:50:01 ixion dovecot[19831]: master: Error: service(anvil): command startup failed, throttling for 2 secs
Oct 26 12:50:01 ixion dovecot[19831]: master: Error: service(ssl-params): command startup failed, throttling for 2 secs


vsz_limit=1024 MB not enough. 2048 MB not enough.

Raised to 3048 MB - now memory is enough but it still isn't able to start
because it hits hardcoded config reading timeout (#define CONFIG_READ_TIMEOUT_SECS 10)

Oct 26 13:00:24 ixion dovecot[20272]: master: Dovecot v2.2.25 (7be1766) starting up for imap, pop3, lmtp (core dumps disabled)
Oct 26 13:00:34 ixion dovecot[20276]: anvil: Fatal: Error reading configuration: Timeout reading config from /var/run/dovecot/config
Oct 26 13:00:34 ixion dovecot[20276]: ssl-params: Fatal: Error reading configuration: Timeout reading config from /var/run/dovecot/config
Oct 26 13:00:34 ixion dovecot[20272]: master: Error: service(anvil): command startup failed, throttling for 2 secs
Oct 26 13:00:34 ixion dovecot[20272]: master: Error: service(ssl-params): command startup failed, throttling for 2 secs


On faster, SSD disk based machine with no load, it starts but then trying to do "doveadm reload"
ends up with timeout
Oct 26 12:22:12 master: Warning: SIGHUP received - reloading configuration
Oct 26 12:22:22 master: Error: Error reading configuration: Timeout reading config from /var/run/dovecot/config


dovecot also seem to stop handling clients when such reload happens.


What can be done to make it work and how?






ps. for comparison on how others handle SSL certificates. Exim MTA for example can easily handle milions of SSL
certificates.That's because it loads certificates on demand, when these are actually needed. Core part of exim ssl
config is two lines:

tls_privatekey =  ${if exists{/etc/certs/cert.${lc:${sha1:${lc:${tls_sni}}}}.pem}{/etc/certs//cert.${lc:${sha1:${lc:${tls_sni}}}}.pem}{/etc/certs/default-cert.key}}
tls_certificate = ${if exists{/etc/certs/cert.${lc:${sha1:${lc:${tls_sni}}}}.pem}{/etc/certs/cert.${lc:${sha1:${lc:${tls_sni}}}}.pem}{/etc/certs/default-cert.pem}}

which means check runtime if file on disk exists; if it exists then use it for tls_XYZ option; if it doesn't exist use default-cert.{pem/key}
(sha1/lowercase game is to make sure that attacker cannot put own filename in TLS SNI servername).

That's all. There is no memory impact and only little cpu impact with such solution. The other big advantage of exim
solution is also no need for daemon to reload its config. You only change/add/remove certificate files on disk.

ps2. thanks to letsencrypt project you can have thousands certificates for each domain/subdomain possible,
so such usage should grow

-- 
Arkadiusz Miśkiewicz, arekm / ( maven.pl | pld-linux.org )