IMAP hibernate and scalability in general

[Christian Balzer]

Hello,

as some may remember, we're running very dense IMAP cluster here, in
excess of 50k IMAP sessions per node (current record holder is 68k, design
is for 200k+).

The first issue we ran into was that the dovecot master process (which is
single thread and thus a bottleneck) was approaching 100% CPU usage (aka
using a full core) when trying to spawn off new IMAP processes.

This was rectified by giving IMAP a service count of 200 to create a pool
of "idling" processes eventually, reducing the strain for the master
process dramatically. That of course required generous cranking up
ulimits, FDs in particular. 

The next issues of course is (as mentioned before) the memory usage of all
those IMAP processes and the fact that quite a few things outside of
dovecote (ps, etc) tend to get quite sedate when dealing with tens of
thousands of processes. 

We just started to deploy a new mailbox cluster pair with 2.2.27 and
having IMAP hibernate configured.
Getting this work is a PITA though with regards to ownership and access
rights to the various sockets, this part could definitely do with some
better (I know, difficult) defaults or at least more documentation (none
besides the source and this ML).

Initial results are very promising, depending on what your clients are
doing (are they well behaved, are your users constantly looking at other
folders, etc) the vast majority of IDLE processes will be in hibernated
at any given time and thus not only using a fraction of the RAM otherwise
needed but also freeing up process space.

Real life example:
240 users, 86 imap processes (80% of those not IDLE) and:
dovecot   119157  0.0  0.0  10452  3236 ?        S    Apr01   0:21 dovecot/imap-hibernate [237 connections]
That's 237 hibernated connections and thus less processes than otherwise.


I assume that given the silence on the ML what we are going to be the
first hibernate users where the term "large scale" does apply. 
Despite that I have some questions, clarifications/confirmations:

Our current default_client_limit is 16k, which can be seen by having 5
config processes on our 65k+ session node. ^_-

This would also apply to imap-hibernate, one wonders if that's fine
(config certainly has no issues) or if something smaller would be
appropriate here?

Since we have idling IMAP processes around most of the time, the strain of
re-spawning proper processes from imap-hibernate should be just as reduced
as for the dovecot master, correct?


I'll keep reporting our experiences here, that is if something blows up
spectacularly. ^o^

Christian
-- 
Christian Balzer        Network/Systems Engineer                
chibi at gol.com   	Global OnLine Japan/Rakuten Communications
http://www.gol.com/