We found out about this through a misconfiguration in our web panel. In the main config file the following line includes a subdirectory;
!include conf.d/*.conf
The web panel created the following two relevant configuration snippets in /etc/dovecot/conf.d/
11-plesk-security-ssl.conf
ssl_dh=</opt/psa/etc/dhparams2048.pem ssl_min_protocol=TLSv1.2 ssl_cipher_list=EECDH+AESGCM+AES128:EECDH+AESGCM+AES256:EECDH+CHACHA20:EECDH+SHA256+AES128:EECDH+SHA384+AES256:EECDH+SHA1+AES128:EECDH+SHA1+AES256:EECDH+HIGH:AESGCM+AES128:AESGCM+AES256:CHACHA20:SHA256+AES128:SHA256+AES256:SHA1+AES128:SHA1+AES256:HIGH:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!3DES:!MD5:!PSK:!KRB5:!aECDH:!kDH:!EDH ssl_prefer_server_ciphers=yes ssl=yes ssl_cert=</etc/dovecot/private/dovecot.pem ssl_key=</etc/dovecot/private/dovecot.pem
This sets a server wide certificate as being the cert in /etc/dovecot/private/dovecot.pem.
Let's say our main server domain is 'acme.com' for example, its 'hostname' (linux hostname) is for example 's02.acme.com'. Then it would also create 14-plesk-sni-mail.acme.com.conf, containing something akin to:
local_name mail.acme.com { ssl_cert = </opt/psa/var/certificates/scf7b9lcY ssl_key = </opt/psa/var/certificates/scf7b9lcY }
These point to the ssl cert and key configured for that specific vhost (not the entire server), which can be different from that configured for the server.
When some clients connected to the server (connecting to mail.acme.com), it would serve the /etc/dovecot/private/dovecot.pem certificate. This was renewed and the clients would work without issue.
Other clients, however, used the certificate at /opt/psa/var/certificates/scf7b9lcY. This certificate is supposed to be the same, but expired and wasn't renewed along with the other one.
This was tricky to figure out since the problems did not appear on our side, only when looking at a client. Once I went around the panel GUI and into the actual dovecot configuration files I found the culprit(s).
I think, but do not know, that it could be related to which IP protocol (V4 or V6) is used to connect to dovecot. All clients who failed happened to connect over V6. The cases that did not used V4.
This seems entirely unrelated and strange behaviour that must be a bug of some kind. Either both should use the one certificate, or the other, or there should be an error or warning if local_name includes a variant subdomain of the server name, the latter in addition to making a consistent choice.
It appears that sometimes the first relevant ssl_key and ssl_cert are used, and other times the second when the option is included twice in the configuration, pointing to different matching certificates/keys.
It's not apparently a race condition: The behaviour is very consistent for the same client connecting.