Hi.
Little story :-)
I'm playing with dovecot 2.2.25 and multiple SSL certificates. ~7000 certificates which are loaded twice, so my dovecot has ~14 000 certificate pairs (14k key + 14k cert) in config.
14 000 local_name entries. Like these:
local_name imap.example.com { ssl_cert = </etc/certs/cert1.pem ssl_key = </etc/certs/cert1.pem }
local_name pop3.example.com { ssl_cert = </etc/certs/cert1.pem ssl_key = </etc/certs/cert1.pem }
(it needs to be "duplicated" like this because dovecot doesn't support globs/regexps for local_name, so can't use: local_name *.example.com { ssl_cert = </etc/certs/cert1.pem ssl_key = </etc/certs/cert1.pem } )
All certificates take 87MB on disk. Dovecot config for certificates itself is 3MB.
Now fun begins. Trying to start dovecot fails
Oct 26 12:49:59 ixion dovecot[19831]: master: Dovecot v2.2.25 (7be1766) starting up for imap, pop3, lmtp (core dumps disabled) Oct 26 12:50:01 ixion dovecot[19835]: config: Fatal: pool_system_malloc(1608): Out of memory Oct 26 12:50:01 ixion dovecot[19831]: master: Error: service(config): command startup failed, throttling for 2 secs Oct 26 12:50:01 ixion dovecot[19835]: config: Fatal: master: service(config): child 19837 returned error 83 (Out of memory (service config { vsz_limit=1024 MB }, you may need to increase it) - set CORE_OUTOFMEM=1 environment to get core dump) Oct 26 12:50:01 ixion dovecot[19835]: anvil: Fatal: Error reading configuration: read(/var/run/dovecot/config) failed: read(size=8192) failed: Connection reset by peer Oct 26 12:50:01 ixion dovecot[19835]: ssl-params: Fatal: Error reading configuration: read(/var/run/dovecot/config) failed: read(size=8192) failed: Connection reset by peer Oct 26 12:50:01 ixion dovecot[19831]: master: Error: service(anvil): command startup failed, throttling for 2 secs Oct 26 12:50:01 ixion dovecot[19831]: master: Error: service(ssl-params): command startup failed, throttling for 2 secs
vsz_limit=1024 MB not enough. 2048 MB not enough.
Raised to 3048 MB - now memory is enough but it still isn't able to start because it hits hardcoded config reading timeout (#define CONFIG_READ_TIMEOUT_SECS 10)
Oct 26 13:00:24 ixion dovecot[20272]: master: Dovecot v2.2.25 (7be1766) starting up for imap, pop3, lmtp (core dumps disabled) Oct 26 13:00:34 ixion dovecot[20276]: anvil: Fatal: Error reading configuration: Timeout reading config from /var/run/dovecot/config Oct 26 13:00:34 ixion dovecot[20276]: ssl-params: Fatal: Error reading configuration: Timeout reading config from /var/run/dovecot/config Oct 26 13:00:34 ixion dovecot[20272]: master: Error: service(anvil): command startup failed, throttling for 2 secs Oct 26 13:00:34 ixion dovecot[20272]: master: Error: service(ssl-params): command startup failed, throttling for 2 secs
On faster, SSD disk based machine with no load, it starts but then trying to do "doveadm reload" ends up with timeout Oct 26 12:22:12 master: Warning: SIGHUP received - reloading configuration Oct 26 12:22:22 master: Error: Error reading configuration: Timeout reading config from /var/run/dovecot/config
dovecot also seem to stop handling clients when such reload happens.
What can be done to make it work and how?
ps. for comparison on how others handle SSL certificates. Exim MTA for example can easily handle milions of SSL certificates.That's because it loads certificates on demand, when these are actually needed. Core part of exim ssl config is two lines:
tls_privatekey = ${if exists{/etc/certs/cert.${lc:${sha1:${lc:${tls_sni}}}}.pem}{/etc/certs//cert.${lc:${sha1:${lc:${tls_sni}}}}.pem}{/etc/certs/default-cert.key}} tls_certificate = ${if exists{/etc/certs/cert.${lc:${sha1:${lc:${tls_sni}}}}.pem}{/etc/certs/cert.${lc:${sha1:${lc:${tls_sni}}}}.pem}{/etc/certs/default-cert.pem}}
which means check runtime if file on disk exists; if it exists then use it for tls_XYZ option; if it doesn't exist use default-cert.{pem/key} (sha1/lowercase game is to make sure that attacker cannot put own filename in TLS SNI servername).
That's all. There is no memory impact and only little cpu impact with such solution. The other big advantage of exim solution is also no need for daemon to reload its config. You only change/add/remove certificate files on disk.
ps2. thanks to letsencrypt project you can have thousands certificates for each domain/subdomain possible, so such usage should grow
-- Arkadiusz Miśkiewicz, arekm / ( maven.pl | pld-linux.org )