IMAP hibernate and scalability in general

[Aki Tuomi]

On 06.04.2017 06:15, Christian Balzer wrote:
> Hello,
>
> as some may remember, we're running very dense IMAP cluster here, in
> excess of 50k IMAP sessions per node (current record holder is 68k, design
> is for 200k+).
>
> The first issue we ran into was that the dovecot master process (which is
> single thread and thus a bottleneck) was approaching 100% CPU usage (aka
> using a full core) when trying to spawn off new IMAP processes.
>
> This was rectified by giving IMAP a service count of 200 to create a pool
> of "idling" processes eventually, reducing the strain for the master
> process dramatically. That of course required generous cranking up
> ulimits, FDs in particular. 
>
> The next issues of course is (as mentioned before) the memory usage of all
> those IMAP processes and the fact that quite a few things outside of
> dovecote (ps, etc) tend to get quite sedate when dealing with tens of
> thousands of processes. 
>
> We just started to deploy a new mailbox cluster pair with 2.2.27 and
> having IMAP hibernate configured.
> Getting this work is a PITA though with regards to ownership and access
> rights to the various sockets, this part could definitely do with some
> better (I know, difficult) defaults or at least more documentation (none
> besides the source and this ML).
>
> Initial results are very promising, depending on what your clients are
> doing (are they well behaved, are your users constantly looking at other
> folders, etc) the vast majority of IDLE processes will be in hibernated
> at any given time and thus not only using a fraction of the RAM otherwise
> needed but also freeing up process space.
>
> Real life example:
> 240 users, 86 imap processes (80% of those not IDLE) and:
> dovecot   119157  0.0  0.0  10452  3236 ?        S    Apr01   0:21 dovecot/imap-hibernate [237 connections]
> That's 237 hibernated connections and thus less processes than otherwise.
>
>
> I assume that given the silence on the ML what we are going to be the
> first hibernate users where the term "large scale" does apply. 
> Despite that I have some questions, clarifications/confirmations:
>
> Our current default_client_limit is 16k, which can be seen by having 5
> config processes on our 65k+ session node. ^_-
>
> This would also apply to imap-hibernate, one wonders if that's fine
> (config certainly has no issues) or if something smaller would be
> appropriate here?
>
> Since we have idling IMAP processes around most of the time, the strain of
> re-spawning proper processes from imap-hibernate should be just as reduced
> as for the dovecot master, correct?
>
>
> I'll keep reporting our experiences here, that is if something blows up
> spectacularly. ^o^
>
> Christian

Hi!

We have customers using it in larger deployments. A good idea is to have
as much of your clients hibernating as possible, as the hibernation
process is much smaller than actual IMAP process.

You should probably also look at reusing the processes, as this will
probably help your performance,
https://wiki.dovecot.org/PerformanceTuning and
https://wiki.dovecot.org/LoginProcess are probably a good starting
point, although I suspect you've read these already.

If you are running a dense server, cranking up various limits is rather
expected.

Aki