IMAP hibernate and scalability in general

[Mark Moseley]

We've been using hibernate for about half a year with no ill effects. There
were various logged errors in earlier versions of dovecot, but even with
those, we never heard a reported customer-side error (almost always when
transitioning from hibernate back to regular imap; in the case of those
errors, presumably the mail client just reconnected silently).

For no particular reason besides wanting to start conservatively, we've got
client_limit set to 50 on the hibernate procs (with 1100 total hibernated
connections on the box I'm looking at). At only a little over a meg each,
I'm fine with those extra processes.

On Wed, Apr 5, 2017 at 11:17 PM, Aki Tuomi <aki.tuomi at dovecot.fi> wrote:

>
>
> On 06.04.2017 06:15, Christian Balzer wrote:
> > Hello,
> >
> > as some may remember, we're running very dense IMAP cluster here, in
> > excess of 50k IMAP sessions per node (current record holder is 68k,
> design
> > is for 200k+).
> >
> > The first issue we ran into was that the dovecot master process (which is
> > single thread and thus a bottleneck) was approaching 100% CPU usage (aka
> > using a full core) when trying to spawn off new IMAP processes.
> >
> > This was rectified by giving IMAP a service count of 200 to create a pool
> > of "idling" processes eventually, reducing the strain for the master
> > process dramatically. That of course required generous cranking up
> > ulimits, FDs in particular.
> >
> > The next issues of course is (as mentioned before) the memory usage of
> all
> > those IMAP processes and the fact that quite a few things outside of
> > dovecote (ps, etc) tend to get quite sedate when dealing with tens of
> > thousands of processes.
> >
> > We just started to deploy a new mailbox cluster pair with 2.2.27 and
> > having IMAP hibernate configured.
> > Getting this work is a PITA though with regards to ownership and access
> > rights to the various sockets, this part could definitely do with some
> > better (I know, difficult) defaults or at least more documentation (none
> > besides the source and this ML).
> >
> > Initial results are very promising, depending on what your clients are
> > doing (are they well behaved, are your users constantly looking at other
> > folders, etc) the vast majority of IDLE processes will be in hibernated
> > at any given time and thus not only using a fraction of the RAM otherwise
> > needed but also freeing up process space.
> >
> > Real life example:
> > 240 users, 86 imap processes (80% of those not IDLE) and:
> > dovecot   119157  0.0  0.0  10452  3236 ?        S    Apr01   0:21
> dovecot/imap-hibernate [237 connections]
> > That's 237 hibernated connections and thus less processes than otherwise.
> >
> >
> > I assume that given the silence on the ML what we are going to be the
> > first hibernate users where the term "large scale" does apply.
> > Despite that I have some questions, clarifications/confirmations:
> >
> > Our current default_client_limit is 16k, which can be seen by having 5
> > config processes on our 65k+ session node. ^_-
> >
> > This would also apply to imap-hibernate, one wonders if that's fine
> > (config certainly has no issues) or if something smaller would be
> > appropriate here?
> >
> > Since we have idling IMAP processes around most of the time, the strain
> of
> > re-spawning proper processes from imap-hibernate should be just as
> reduced
> > as for the dovecot master, correct?
> >
> >
> > I'll keep reporting our experiences here, that is if something blows up
> > spectacularly. ^o^
> >
> > Christian
>
> Hi!
>
> We have customers using it in larger deployments. A good idea is to have
> as much of your clients hibernating as possible, as the hibernation
> process is much smaller than actual IMAP process.
>
> You should probably also look at reusing the processes, as this will
> probably help your performance,
> https://wiki.dovecot.org/PerformanceTuning and
> https://wiki.dovecot.org/LoginProcess are probably a good starting
> point, although I suspect you've read these already.
>
> If you are running a dense server, cranking up various limits is rather
> expected.
>
> Aki
>