System load spike on dovecot reload

[Christian Balzer]

Hello,

On Fri, 21 Apr 2017 10:43:47 +0200 dave at evilcigi.eu wrote:

> Hi everyone,
> 
> I'm running dovecot with quite a lot of users and lots of active imap 
> connections (like 20'000). I'm using different user IDs for users, so I 
> need to have imap {service_count=1} - i.e. I have a lots of imap 
> processes running.
>
We peaked out at 65k imap processes before upgrading to a version where
imap-hibernate more or less works, but we're using a common ID.
---
dovecot   119157  0.1  0.0  59364 52216 ?        S    Apr01  48:25 dovecot/imap-hibernate [15137 connections]
---

The service_count parameter in this context is not doing what you think it
does, I have it at 200 these days and that will allow imap (or pop3)
processes to be recycled (they are labeled with "idling" when waiting for a
new client), not having one imap process serve multiple clients. 
---
mail      591307  0.0  0.0  29876  4712 ?        S    Apr20   0:00 dovecot/imap [idling]
mail      735323  0.0  0.0  27396  4196 ?        S    13:20   0:00 dovecot/pop3 [idling]
---

The advantage (for me at least) is that the dovecot master process doesn't
have to to spin up a new mail processes each time during logins.

Since this process is quite single-threaded, it becomes a bottleneck
eventually.
  
> Everything works fine, until I reload dovecot configuration. When that 
> happen, every client is forced to relogin in the same time and that 
> causes a huge system load spike (2-3000 5 min load).
> 
Unless you're making a change that affects the dovecot master process,
restarting everything isn't needed and you should set 
"shutdown_clients = no". 
You could still kick users with "dovecot kick" at a leisurely pace, but
security problems with the mail processes are rare.

> I was thinking that it would be great, if dovecot wouldn't kick all the 
> users in the same time during reload, but somehow gradually, during 
> specified interval. I'm aware of the shutdown_clients directive that 
> could help, but I don't like it - 

I've very much gotten to like it, once things got huge and busy.

> I do want the clients get disconnected 
> on dovecot shutdown and also I want them to relogin in reasonably short 
> time after reload.
> 
> Is something like that possible with dovecot or does it make sense to 
> implement that in the future versions?
> 
Run a dovecot proxy (if you have single box with all these users on it,
Mr. Murphy would like a word with you) and set 
"login_proxy_max_disconnect_delay" to something that suits you.

Christian
-- 
Christian Balzer        Network/Systems Engineer                
chibi at gol.com   	Global OnLine Japan/Rakuten Communications
http://www.gol.com/