Hello,
On Wed, 5 Apr 2017 23:45:33 -0700 Mark Moseley wrote:
We've been using hibernate for about half a year with no ill effects. There were various logged errors in earlier versions of dovecot, but even with those, we never heard a reported customer-side error (almost always when transitioning from hibernate back to regular imap; in the case of those errors, presumably the mail client just reconnected silently).
This is my impression as well (silent reconnect w/o anything really bad happening) with regard to the bug I just reported/saw.
For no particular reason besides wanting to start conservatively, we've got client_limit set to 50 on the hibernate procs (with 1100 total hibernated connections on the box I'm looking at). At only a little over a meg each, I'm fine with those extra processes.
Yeah, but 50 would be a tad too conservative for our purposes here. I'll keep an eye on it and see how it goes, first checkpoint would be at 1k hibernated sessions. ^_^
Christian
On Wed, Apr 5, 2017 at 11:17 PM, Aki Tuomi aki.tuomi@dovecot.fi wrote:
On 06.04.2017 06:15, Christian Balzer wrote:
Hello,
as some may remember, we're running very dense IMAP cluster here, in excess of 50k IMAP sessions per node (current record holder is 68k,
design
is for 200k+).The first issue we ran into was that the dovecot master process (which is single thread and thus a bottleneck) was approaching 100% CPU usage (aka using a full core) when trying to spawn off new IMAP processes.
This was rectified by giving IMAP a service count of 200 to create a pool of "idling" processes eventually, reducing the strain for the master process dramatically. That of course required generous cranking up ulimits, FDs in particular.
The next issues of course is (as mentioned before) the memory usage of
all
those IMAP processes and the fact that quite a few things outside of dovecote (ps, etc) tend to get quite sedate when dealing with tens of thousands of processes.We just started to deploy a new mailbox cluster pair with 2.2.27 and having IMAP hibernate configured. Getting this work is a PITA though with regards to ownership and access rights to the various sockets, this part could definitely do with some better (I know, difficult) defaults or at least more documentation (none besides the source and this ML).
Initial results are very promising, depending on what your clients are doing (are they well behaved, are your users constantly looking at other folders, etc) the vast majority of IDLE processes will be in hibernated at any given time and thus not only using a fraction of the RAM otherwise needed but also freeing up process space.
Real life example: 240 users, 86 imap processes (80% of those not IDLE) and: dovecot 119157 0.0 0.0 10452 3236 ? S Apr01 0:21
dovecot/imap-hibernate [237 connections]
That's 237 hibernated connections and thus less processes than otherwise.I assume that given the silence on the ML what we are going to be the first hibernate users where the term "large scale" does apply. Despite that I have some questions, clarifications/confirmations:
Our current default_client_limit is 16k, which can be seen by having 5 config processes on our 65k+ session node. ^_-
This would also apply to imap-hibernate, one wonders if that's fine (config certainly has no issues) or if something smaller would be appropriate here?
Since we have idling IMAP processes around most of the time, the strain
of
re-spawning proper processes from imap-hibernate should be just as
reduced
as for the dovecot master, correct?I'll keep reporting our experiences here, that is if something blows up spectacularly. ^o^
Christian
Hi!
We have customers using it in larger deployments. A good idea is to have as much of your clients hibernating as possible, as the hibernation process is much smaller than actual IMAP process.
You should probably also look at reusing the processes, as this will probably help your performance, https://wiki.dovecot.org/PerformanceTuning and https://wiki.dovecot.org/LoginProcess are probably a good starting point, although I suspect you've read these already.
If you are running a dense server, cranking up various limits is rather expected.
Aki
--
Christian Balzer Network/Systems Engineer
chibi@gol.com Global OnLine Japan/Rakuten Communications
http://www.gol.com/