[Dovecot] service_count=0 for imap and pop3
This is copy-pasted from Timo's comment in the "Todays Performance Data for 2.0.x" thread, but I didn't want to hijack that thread. In it, Timo says:
<snip> service imap { service_count = 0 }
(i.e. reuse imap processes) reduced the system CPU usage to almost nothing. But if you use different UIDs for different users you can't do that. </snip>
Is that safe to do in imap and/or pop3? Or at least no more insecure than using service_count=0 for imap-login and pop3-login? Since I saw that comment I've been googling around and grepping the wiki but I can only find instances of using it with "service imap-login", not "service imap" or "service pop3".
The mention in Timo's comment that it reduces the CPU usage by a lot got my attention, esp for pop3. We're already using service_count=0 for pop3-login and imap-login.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Le 10.11.2010 00:05, Timo Sirainen a écrit :
On 9.11.2010, at 22.14, Mark Moseley wrote:
service imap { service_count = 0 }
Is that safe to do in imap and/or pop3? Or at least no more insecure than using service_count=0 for imap-login and pop3-login?
Yep.
Timo, I'm quite missed us I find the answer "yep" short. we should do what exactly. edited the wiki, Timo
<anonymous>
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x092164A7 gpg --keyserver pgp.mit.edu --recv-key 092164A7 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (GNU/Linux) Comment: Using GnuPG with CentOS - http://enigmail.mozdev.org/
iD8DBQFM2dsMtXI/OwkhZKcRAoF9AJ0SoRPEDvcg8v0MpKCWSiRjbCqILgCfZDgg Qx5Lb230c1tNkAsL0GTC508= =lwux -----END PGP SIGNATURE-----
On Tue, Nov 9, 2010 at 3:05 PM, Timo Sirainen <tss@iki.fi> wrote:
On 9.11.2010, at 22.14, Mark Moseley wrote:
service imap { service_count = 0 }
Is that safe to do in imap and/or pop3? Or at least no more insecure than using service_count=0 for imap-login and pop3-login?
Yep.
Would the risks involved be identical to your warnings about using "service_count=0" with pop3-login/imap-login, namely that if the daemon gets hacked, it'd be able to access other mailboxes (presumably that the imap/pop3 already had open)?
Nice, it does indeed seem to burn a lot less CPU. I've also set "process_min_avail=#" for 'service pop3' which appears to spread out incoming POP3 connections over all # pop3 procs. Any gotchas there? I've always got at least several hundred POP3 connections on a box, so having them not all hitting one proc is desirable. And having, say, 100 pop3 procs hanging around, possibly idle, is fine. This is pretty exciting stuff.
Anybody running this way in production on a large scale, i.e. using "service_count=0" in a "service imap" or "service pop3" ?
On 9.11.2010, at 23.49, Mark Moseley wrote:
On Tue, Nov 9, 2010 at 3:05 PM, Timo Sirainen <tss@iki.fi> wrote:
On 9.11.2010, at 22.14, Mark Moseley wrote:
service imap { service_count = 0 }
Would the risks involved be identical to your warnings about using "service_count=0" with pop3-login/imap-login, namely that if the daemon gets hacked, it'd be able to access other mailboxes (presumably that the imap/pop3 already had open)?
If all your users share the same UID and there was a security hole in imap/pop3, you could access everyone's mails regardless of this setting. (Except if you also enabled chroots, but then you couldn't use this setting.)
Nice, it does indeed seem to burn a lot less CPU. I've also set "process_min_avail=#" for 'service pop3' which appears to spread out incoming POP3 connections over all # pop3 procs. Any gotchas there? I've always got at least several hundred POP3 connections on a box, so having them not all hitting one proc is desirable. And having, say, 100 pop3 procs hanging around, possibly idle, is fine. This is pretty exciting stuff.
Should be fine.
Anybody running this way in production on a large scale, i.e. using "service_count=0" in a "service imap" or "service pop3" ?
Only potential problem is memory leaks that keep increasing the memory usage. Of course there should be no memory leaks. :) You could anyway set something like service_count=1000 to get it to restart after handling 1000 connections.
On Tue, Nov 9, 2010 at 3:55 PM, Timo Sirainen <tss@iki.fi> wrote:
On 9.11.2010, at 23.49, Mark Moseley wrote:
On Tue, Nov 9, 2010 at 3:05 PM, Timo Sirainen <tss@iki.fi> wrote:
On 9.11.2010, at 22.14, Mark Moseley wrote:
service imap { service_count = 0 }
Would the risks involved be identical to your warnings about using "service_count=0" with pop3-login/imap-login, namely that if the daemon gets hacked, it'd be able to access other mailboxes (presumably that the imap/pop3 already had open)?
If all your users share the same UID and there was a security hole in imap/pop3, you could access everyone's mails regardless of this setting. (Except if you also enabled chroots, but then you couldn't use this setting.)
Nice, it does indeed seem to burn a lot less CPU. I've also set "process_min_avail=#" for 'service pop3' which appears to spread out incoming POP3 connections over all # pop3 procs. Any gotchas there? I've always got at least several hundred POP3 connections on a box, so having them not all hitting one proc is desirable. And having, say, 100 pop3 procs hanging around, possibly idle, is fine. This is pretty exciting stuff.
Should be fine.
Anybody running this way in production on a large scale, i.e. using "service_count=0" in a "service imap" or "service pop3" ?
Only potential problem is memory leaks that keep increasing the memory usage. Of course there should be no memory leaks. :) You could anyway set something like service_count=1000 to get it to restart after handling 1000 connections.
I'll keep that one in mind. Doesn't seem like it could hurt to have something like =1000 just to be safe.
For anybody else reading this thread, here's vmstat from two boxes with an almost identical amount of IMAP and POP3 connections, but the first example has "service_count=0" for "service pop3", the second doesn't. Check out the context switches (this will paste horribly; it's the 5th column from the right):
With "service_count=0": r b swpd free buff cache si so bi bo in cs us sy id wa 0 0 0 54056 161536 1116012 0 0 132 0 1326 7868 3 8 85 4 0 0 0 53568 161608 1116192 0 0 116 0 1572 5111 2 6 80 13 1 0 0 52320 161608 1116960 0 0 16 140 1584 6126 3 6 86 5 0 0 0 51940 161656 1117292 0 0 44 128 1430 5508 3 7 86 5 0 0 0 53316 161628 1115500 0 0 12 0 1459 5363 4 7 86 4 0 0 0 52176 161664 1115468 0 0 76 76 1221 6344 6 5 84 6 0 1 0 52300 161688 1115636 0 0 32 0 1250 3631 3 2 90 6 0 0 0 53580 161648 1113116 0 0 76 172 1778 6671 3 9 83 6
Without "service_count=0": r b swpd free buff cache si so bi bo in cs us sy id wa 0 2 2476 73776 147860 958728 0 0 0 0 912 17442 6 11 0 84 0 2 2476 61300 147860 958756 0 0 0 0 992 22130 5 19 0 76 21 4 2476 49484 148008 957608 0 0 140 748 2472 15922 9 24 0 68 30 4 2476 97928 146604 945232 0 0 404 0 7116 20700 43 58 0 0 4 1 2476 169280 146700 946972 0 0 172 0 7014 43727 33 67 1 0 7 1 2476 168684 146800 947304 0 0 168 0 5720 55485 37 62 2 1 9 0 2476 165956 146856 947504 0 0 80 0 5074 56371 35 65 0 0 6 6 2476 163032 146948 948380 0 0 92 2432 5879 63418 27 67 5 2
It's nice to look at 'perf top' on the first box and not see "finish_task_switch" chewing up 15+% of kernel CPU time :)
Only potential problem is memory leaks that keep increasing the memory usage. Of course there should be no memory leaks. :) You could anyway set something like service_count=1000 to get it to restart after handling 1000 connections.
I'll keep that one in mind. Doesn't seem like it could hurt to have something like =1000 just to be safe.
For anybody else reading this thread, here's vmstat from two boxes with an almost identical amount of IMAP and POP3 connections, but the first example has "service_count=0" for "service pop3", the second doesn't. Check out the context switches (this will paste horribly; it's the 5th column from the right):
With "service_count=0": r b swpd free buff cache si so bi bo in cs us sy id wa 0 0 0 54056 161536 1116012 0 0 132 0 1326 7868 3 8 85 4 0 0 0 53568 161608 1116192 0 0 116 0 1572 5111 2 6 80 13 1 0 0 52320 161608 1116960 0 0 16 140 1584 6126 3 6 86 5 0 0 0 51940 161656 1117292 0 0 44 128 1430 5508 3 7 86 5 0 0 0 53316 161628 1115500 0 0 12 0 1459 5363 4 7 86 4 0 0 0 52176 161664 1115468 0 0 76 76 1221 6344 6 5 84 6 0 1 0 52300 161688 1115636 0 0 32 0 1250 3631 3 2 90 6 0 0 0 53580 161648 1113116 0 0 76 172 1778 6671 3 9 83 6
Without "service_count=0": r b swpd free buff cache si so bi bo in cs us sy id wa 0 2 2476 73776 147860 958728 0 0 0 0 912 17442 6 11 0 84 0 2 2476 61300 147860 958756 0 0 0 0 992 22130 5 19 0 76 21 4 2476 49484 148008 957608 0 0 140 748 2472 15922 9 24 0 68 30 4 2476 97928 146604 945232 0 0 404 0 7116 20700 43 58 0 0 4 1 2476 169280 146700 946972 0 0 172 0 7014 43727 33 67 1 0 7 1 2476 168684 146800 947304 0 0 168 0 5720 55485 37 62 2 1 9 0 2476 165956 146856 947504 0 0 80 0 5074 56371 35 65 0 0 6 6 2476 163032 146948 948380 0 0 92 2432 5879 63418 27 67 5 2
It's nice to look at 'perf top' on the first box and not see "finish_task_switch" chewing up 15+% of kernel CPU time :)
Timo, Any hints on how many POP3 and IMAP connections I'd be able to get away with in a single threads with the above setup, assuming they're relative busy? I.e. if my boxes typically have, say, 200 concurrent POP3 connections and 600 IMAP connections, if I used process_min_avail=50 for POP3 and process_min_avail=100 for IMAP, I'd assume that'd be in the area of 4 POP3 concurrent connections handled by that one POP3 thread and 6 concurrent connections handled by that one IMAP thead.
I was thinking something like 3-4 per thread (calculating vs max connections) for POP3 and maybe 5-7 for IMAP (as I'm assuming that at least a few will just be in IDLE at any given point).
I guess it comes down to, are you aware of any internal limits for a POP3 or IMAP thread where they're going to start to bog down and possibly death spiral? I'm going to experiment with different settings, but I'd love to hear any opinions you have on it. Sorry for the vagueness.
BTW, I realize process_min_avail is just a minimum, but without having service_count=1, what would cause dovecot to fork off more processes than #process_min_avail?
Thanks!
Timo, Any hints on how many POP3 and IMAP connections I'd be able to get away with in a single threads with the above setup, assuming they're relative busy? I.e. if my boxes typically have, say, 200 concurrent POP3 connections and 600 IMAP connections, if I used process_min_avail=50 for POP3 and process_min_avail=100 for IMAP, I'd assume that'd be in the area of 4 POP3 concurrent connections handled by that one POP3 thread and 6 concurrent connections handled by that one IMAP thead.
I was thinking something like 3-4 per thread (calculating vs max connections) for POP3 and maybe 5-7 for IMAP (as I'm assuming that at least a few will just be in IDLE at any given point).
I guess it comes down to, are you aware of any internal limits for a POP3 or IMAP thread where they're going to start to bog down and possibly death spiral? I'm going to experiment with different settings, but I'd love to hear any opinions you have on it. Sorry for the vagueness.
BTW, I realize process_min_avail is just a minimum, but without having service_count=1, what would cause dovecot to fork off more processes than #process_min_avail?
Actually, it looks like the "spreading" of connections was in my imagination only :) It looks like it can get quite 'clumpy'. Here's a copy/paste from a small segment of ps:
emailuser 2625 0.0 0.1 4748 3132 ? S 17:15 0:00 \_ dovecot/imap [2 connections] emailuser 2626 0.0 0.2 7484 6124 ? S 17:15 0:00 \_ dovecot/imap [16 connections] emailuser 2627 0.0 0.1 4572 2960 ? S 17:15 0:00 \_ dovecot/imap [5 connections] emailuser 2628 0.0 0.5 12968 10332 ? S 17:15 0:01 \_ dovecot/imap [30 connections] emailuser 2630 0.2 1.0 23316 20836 ? S 17:15 0:04 \_ dovecot/imap [90 connections] emailuser 2632 0.7 2.6 59440 54492 ? S 17:15 0:14 \_ dovecot/imap [209 connections] emailuser 4099 0.7 0.3 8636 7252 ? S 17:34 0:06 \_ dovecot/pop3 [2 connections] emailuser 4418 0.8 0.3 9576 8196 ? S 17:38 0:05 \_ dovecot/pop3 [6 connections] emailuser 4478 0.5 0.7 18664 14868 ? S 17:39 0:03 \_ dovecot/imap [62 connections] emailuser 4622 1.1 0.4 10404 8780 ? S 17:40 0:05 \_ dovecot/pop3 [8 connections] emailuser 4733 1.1 0.3 8576 6972 ? S 17:42 0:04 \_ dovecot/pop3 [5 connections]
The interesting thing is that there's quite a number of other imap/pop3 procs with just one connection and a ton of imap procs that haven't been touched (i.e. are still running as 'root', waiting to eventually setuid to 'emailuser').
Is there a way to spread those out more or is there probably no need to? I imagine in the case of PID 2632, a large number of those connections are just sitting in IDLE and doing nothing beyond stat64()'s. Or maybe a better question would be, is there a setting I'm not finding that puts an upper limit on connections for a single process, so a single process will stop servicing new connections after it hits a certain # and lets the other less-loaded processes handle the new connection. All the various *_limit settings appear to be across all processes, not per-processs like this. This btw is 2.0.7 (but with the proctitle patch).
On 15.11.2010, at 22.58, Mark Moseley wrote:
Timo, Any hints on how many POP3 and IMAP connections I'd be able to get away with in a single threads with the above setup, assuming they're relative busy?
The problem is that if there is any waiting for locks, all the other connections hang there as well waiting for it. Same for any type of disk I/O waits. I don't really know how many you could get away with.. My official recommendation would be to keep it 1 connection = 1 process, since that's guaranteed to work.
BTW, I realize process_min_avail is just a minimum, but without having service_count=1, what would cause dovecot to fork off more processes than #process_min_avail?
When all the existing processes have reached client_limit, a new process is created.
emailuser 2625 0.0 0.1 4748 3132 ? S 17:15 0:00 \_ dovecot/imap [2 connections] .. emailuser 2632 0.7 2.6 59440 54492 ? S 17:15 0:14 \_ dovecot/imap [209 connections]
Is there a way to spread those out more or is there probably no need to?
Whichever process is fastest to grab a new connection gets it.
I imagine in the case of PID 2632, a large number of those connections are just sitting in IDLE and doing nothing beyond stat64()'s. Or maybe a better question would be, is there a setting I'm not finding that puts an upper limit on connections for a single process, so a single process will stop servicing new connections after it hits a certain # and lets the other less-loaded processes handle the new connection. All the various *_limit settings appear to be across all processes, not per-processs like this. This btw is 2.0.7 (but with the proctitle patch).
service imap { client_limit = 5 }
Timo, Any hints on how many POP3 and IMAP connections I'd be able to get away with in a single threads with the above setup, assuming they're relative busy?
The problem is that if there is any waiting for locks, all the other connections hang there as well waiting for it. Same for any type of disk I/O waits. I don't really know how many you could get away with.. My official recommendation would be to keep it 1 connection = 1 process, since that's guaranteed to work.
BTW, I realize process_min_avail is just a minimum, but without having service_count=1, what would cause dovecot to fork off more processes than #process_min_avail?
When all the existing processes have reached client_limit, a new process is created.
emailuser 2625 0.0 0.1 4748 3132 ? S 17:15 0:00 \_ dovecot/imap [2 connections] .. emailuser 2632 0.7 2.6 59440 54492 ? S 17:15 0:14 \_ dovecot/imap [209 connections]
Is there a way to spread those out more or is there probably no need to?
Whichever process is fastest to grab a new connection gets it.
I imagine in the case of PID 2632, a large number of those connections are just sitting in IDLE and doing nothing beyond stat64()'s. Or maybe a better question would be, is there a setting I'm not finding that puts an upper limit on connections for a single process, so a single process will stop servicing new connections after it hits a certain # and lets the other less-loaded processes handle the new connection. All the various *_limit settings appear to be across all processes, not per-processs like this. This btw is 2.0.7 (but with the proctitle patch).
service imap { client_limit = 5 }
Yeah, client_limit looks perfect. I'd googled "client_limit" a few times over the past week and for some reason, I thought client_limit was also cumulative across all procs, instead of per-proc. But it's working perfectly now.
I see your point about any blocking operations. I'll start low and work my way up and abandon service_limit=0 completely if I start to get user complaints about it. Right now the possibility of cutting context switches by 80% is just too promising to pass up. With typically 600-700 imap procs and 200-300 pop3 procs, the time spent in kernel scheduling was getting very high and these are only dual core uniprocessor boxes, otherwise it'd be more of a moot point if I had 8+ cores to work with.
Thanks Timo!
participants (3)
-
fakessh ＠
-
Mark Moseley
-
Timo Sirainen