Thank you Joseph and Aki!
You got it right, the issue was indeed with this service_count=100. With service_count=0 it works as intended (only 4 imap-login processes), though now we're concerned about possible memory leaks with this config.
What you described Jospeh (https://www.mail-archive.com/dovecot%40dovecot.org/msg85850.html) is what we've observed as well. In addition, service_count > 1 + high process_limit consumes much more mermory because of all those imap-login processes handling juste a few lasting connections. We're consuming about 4x less memory with service_count=0, it's day and night.
There's something somewhat close documented on https://doc.dovecot.org/configuration_manual/service_configuration/#service-...:
"Otherwise when the service_count is beginning to be reached, the total number of available connections will shrink. With very bad luck that could mean that all the processes are simply waiting for the existing connections to die away before the process can die and a new one can be created. "
Though not the focus of the discussion, it does say that processes don't die until their connections have died.
It could perhaps benefit from mentioning a few more things like:
- service_count = 0 has no protection against potential memory leaks.
- service_count > 1 + high process_limit coud produce many processes since these don't actually die until their connections have all died, which consumes isgnificantly more memory.
One workaround to the lack of memory leaks protection could be to set process_limit close to process_min_avail while keeping service_count > 1. But we'd end up in the risky case described in the docs:
"With very bad luck that could mean that all the processes are simply waiting for the existing connections to die away before the process can die and a new one can be created."
So for now we don't see any way out of service_count = 0 and it's associated memory leak risk.