IMAP hibernate and scalability in general

[Christian Balzer]

Hello,

On Wed, 5 Apr 2017 23:45:33 -0700 Mark Moseley wrote:

> We've been using hibernate for about half a year with no ill effects. There
> were various logged errors in earlier versions of dovecot, but even with
> those, we never heard a reported customer-side error (almost always when
> transitioning from hibernate back to regular imap; in the case of those
> errors, presumably the mail client just reconnected silently).
> 
This is my impression as well (silent reconnect w/o anything really bad
happening) with regard to the bug I just reported/saw. 

> For no particular reason besides wanting to start conservatively, we've got
> client_limit set to 50 on the hibernate procs (with 1100 total hibernated
> connections on the box I'm looking at). At only a little over a meg each,
> I'm fine with those extra processes.
> 
Yeah, but 50 would be a tad too conservative for our purposes here.
I'll keep an eye on it and see how it goes, first checkpoint would be at
1k hibernated sessions. ^_^

Christian

> On Wed, Apr 5, 2017 at 11:17 PM, Aki Tuomi <aki.tuomi at dovecot.fi> wrote:
> 
> >
> >
> > On 06.04.2017 06:15, Christian Balzer wrote:  
> > > Hello,
> > >
> > > as some may remember, we're running very dense IMAP cluster here, in
> > > excess of 50k IMAP sessions per node (current record holder is 68k,  
> > design  
> > > is for 200k+).
> > >
> > > The first issue we ran into was that the dovecot master process (which is
> > > single thread and thus a bottleneck) was approaching 100% CPU usage (aka
> > > using a full core) when trying to spawn off new IMAP processes.
> > >
> > > This was rectified by giving IMAP a service count of 200 to create a pool
> > > of "idling" processes eventually, reducing the strain for the master
> > > process dramatically. That of course required generous cranking up
> > > ulimits, FDs in particular.
> > >
> > > The next issues of course is (as mentioned before) the memory usage of  
> > all  
> > > those IMAP processes and the fact that quite a few things outside of
> > > dovecote (ps, etc) tend to get quite sedate when dealing with tens of
> > > thousands of processes.
> > >
> > > We just started to deploy a new mailbox cluster pair with 2.2.27 and
> > > having IMAP hibernate configured.
> > > Getting this work is a PITA though with regards to ownership and access
> > > rights to the various sockets, this part could definitely do with some
> > > better (I know, difficult) defaults or at least more documentation (none
> > > besides the source and this ML).
> > >
> > > Initial results are very promising, depending on what your clients are
> > > doing (are they well behaved, are your users constantly looking at other
> > > folders, etc) the vast majority of IDLE processes will be in hibernated
> > > at any given time and thus not only using a fraction of the RAM otherwise
> > > needed but also freeing up process space.
> > >
> > > Real life example:
> > > 240 users, 86 imap processes (80% of those not IDLE) and:
> > > dovecot   119157  0.0  0.0  10452  3236 ?        S    Apr01   0:21  
> > dovecot/imap-hibernate [237 connections]  
> > > That's 237 hibernated connections and thus less processes than otherwise.
> > >
> > >
> > > I assume that given the silence on the ML what we are going to be the
> > > first hibernate users where the term "large scale" does apply.
> > > Despite that I have some questions, clarifications/confirmations:
> > >
> > > Our current default_client_limit is 16k, which can be seen by having 5
> > > config processes on our 65k+ session node. ^_-
> > >
> > > This would also apply to imap-hibernate, one wonders if that's fine
> > > (config certainly has no issues) or if something smaller would be
> > > appropriate here?
> > >
> > > Since we have idling IMAP processes around most of the time, the strain  
> > of  
> > > re-spawning proper processes from imap-hibernate should be just as  
> > reduced  
> > > as for the dovecot master, correct?
> > >
> > >
> > > I'll keep reporting our experiences here, that is if something blows up
> > > spectacularly. ^o^
> > >
> > > Christian  
> >
> > Hi!
> >
> > We have customers using it in larger deployments. A good idea is to have
> > as much of your clients hibernating as possible, as the hibernation
> > process is much smaller than actual IMAP process.
> >
> > You should probably also look at reusing the processes, as this will
> > probably help your performance,
> > https://wiki.dovecot.org/PerformanceTuning and
> > https://wiki.dovecot.org/LoginProcess are probably a good starting
> > point, although I suspect you've read these already.
> >
> > If you are running a dense server, cranking up various limits is rather
> > expected.
> >
> > Aki
> >  
> 


-- 
Christian Balzer        Network/Systems Engineer                
chibi at gol.com   	Global OnLine Japan/Rakuten Communications
http://www.gol.com/