On 06.04.2017 06:15, Christian Balzer wrote:
Hello,
as some may remember, we're running very dense IMAP cluster here, in excess of 50k IMAP sessions per node (current record holder is 68k, design is for 200k+).
The first issue we ran into was that the dovecot master process (which is single thread and thus a bottleneck) was approaching 100% CPU usage (aka using a full core) when trying to spawn off new IMAP processes.
This was rectified by giving IMAP a service count of 200 to create a pool of "idling" processes eventually, reducing the strain for the master process dramatically. That of course required generous cranking up ulimits, FDs in particular.
The next issues of course is (as mentioned before) the memory usage of all those IMAP processes and the fact that quite a few things outside of dovecote (ps, etc) tend to get quite sedate when dealing with tens of thousands of processes.
We just started to deploy a new mailbox cluster pair with 2.2.27 and having IMAP hibernate configured. Getting this work is a PITA though with regards to ownership and access rights to the various sockets, this part could definitely do with some better (I know, difficult) defaults or at least more documentation (none besides the source and this ML).
Initial results are very promising, depending on what your clients are doing (are they well behaved, are your users constantly looking at other folders, etc) the vast majority of IDLE processes will be in hibernated at any given time and thus not only using a fraction of the RAM otherwise needed but also freeing up process space.
Real life example: 240 users, 86 imap processes (80% of those not IDLE) and: dovecot 119157 0.0 0.0 10452 3236 ? S Apr01 0:21 dovecot/imap-hibernate [237 connections] That's 237 hibernated connections and thus less processes than otherwise.
I assume that given the silence on the ML what we are going to be the first hibernate users where the term "large scale" does apply. Despite that I have some questions, clarifications/confirmations:
Our current default_client_limit is 16k, which can be seen by having 5 config processes on our 65k+ session node. ^_-
This would also apply to imap-hibernate, one wonders if that's fine (config certainly has no issues) or if something smaller would be appropriate here?
Since we have idling IMAP processes around most of the time, the strain of re-spawning proper processes from imap-hibernate should be just as reduced as for the dovecot master, correct?
I'll keep reporting our experiences here, that is if something blows up spectacularly. ^o^
Christian
Hi!
We have customers using it in larger deployments. A good idea is to have as much of your clients hibernating as possible, as the hibernation process is much smaller than actual IMAP process.
You should probably also look at reusing the processes, as this will probably help your performance, https://wiki.dovecot.org/PerformanceTuning and https://wiki.dovecot.org/LoginProcess are probably a good starting point, although I suspect you've read these already.
If you are running a dense server, cranking up various limits is rather expected.
Aki