[Dovecot] dovecot performance
Hello All,
I've been studying dovecot for replacing my company's current system and I got a little worried about an aspect of the dovecot's design. I was surprised that dovecot doesn't use prefork for its mail processes, forking a new processes for each new client connection.
Talking in the #dovecot channel I was gave a scenario of a system supporting ~40k users with 4 servers just fine. I wonder how well dovecot would scale if we increase this number of users by some order of magnitude like, say, 4M users.
It is well known that preforking is a good pratice if you want to achieve a higher performance. When I was asked about it I readily answered: "of course it does". For my surprise later, i doesn't.
Do you have any plans to support preforking in the near future?
Best regards, giorgenes
Words by Giorgenes Gelatti [Thu, Aug 14, 2008 at 03:38:50PM -0300]:
Hello All,
I've been studying dovecot for replacing my company's current system and I got a little worried about an aspect of the dovecot's design. I was surprised that dovecot doesn't use prefork for its mail processes, forking a new processes for each new client connection.
Talking in the #dovecot channel I was gave a scenario of a system supporting ~40k users with 4 servers just fine. I wonder how well dovecot would scale if we increase this number of users by some order of magnitude like, say, 4M users.
Well, we have 8 servers for that amount of users.
It is well known that preforking is a good pratice if you want to achieve a higher performance.
Some say it's premature optimization.
When I was asked about it I readily answered: "of course it does". For my surprise later, i doesn't.
Do you have any plans to support preforking in the near future?
-- Jose Celestino | http://japc.uncovering.org/files/japc-pgpkey.asc
"One man’s theology is another man’s belly laugh." -- Robert A. Heinlein
Woa!!
Do you have statistics of access/min for pop3?
Indeed it could be premature since I didn't measure any real bottleneck. Just something that got my attention.
[]'s giorgenes
2008/8/14 Jose Celestino japc@co.sapo.pt:
Words by Giorgenes Gelatti [Thu, Aug 14, 2008 at 03:38:50PM -0300]:
Hello All,
I've been studying dovecot for replacing my company's current system and I got a little worried about an aspect of the dovecot's design. I was surprised that dovecot doesn't use prefork for its mail processes, forking a new processes for each new client connection.
Talking in the #dovecot channel I was gave a scenario of a system supporting ~40k users with 4 servers just fine. I wonder how well dovecot would scale if we increase this number of users by some order of magnitude like, say, 4M users.
Well, we have 8 servers for that amount of users.
It is well known that preforking is a good pratice if you want to achieve a higher performance.
Some say it's premature optimization.
When I was asked about it I readily answered: "of course it does". For my surprise later, i doesn't.
Do you have any plans to support preforking in the near future?
-- Jose Celestino | http://japc.uncovering.org/files/japc-pgpkey.asc
"One man's theology is another man's belly laugh." -- Robert A. Heinlein
Words by Giorgenes Gelatti [Thu, Aug 14, 2008 at 04:46:04PM -0300]:
Woa!!
Do you have statistics of access/min for pop3?
No, but I 'greped' 2300 logins/minute peak for a given day for *IMAP*.
Indeed it could be premature since I didn't measure any real bottleneck. Just something that got my attention.
[]'s giorgenes
2008/8/14 Jose Celestino japc@co.sapo.pt:
Words by Giorgenes Gelatti [Thu, Aug 14, 2008 at 03:38:50PM -0300]:
Hello All,
I've been studying dovecot for replacing my company's current system and I got a little worried about an aspect of the dovecot's design. I was surprised that dovecot doesn't use prefork for its mail processes, forking a new processes for each new client connection.
Talking in the #dovecot channel I was gave a scenario of a system supporting ~40k users with 4 servers just fine. I wonder how well dovecot would scale if we increase this number of users by some order of magnitude like, say, 4M users.
Well, we have 8 servers for that amount of users.
It is well known that preforking is a good pratice if you want to achieve a higher performance.
Some say it's premature optimization.
When I was asked about it I readily answered: "of course it does". For my surprise later, i doesn't.
Do you have any plans to support preforking in the near future?
-- Jose Celestino | http://japc.uncovering.org/files/japc-pgpkey.asc
"One man’s theology is another man’s belly laugh." -- Robert A. Heinlein
On Aug 14, 2008, at 2:38 PM, Giorgenes Gelatti wrote:
I've been studying dovecot for replacing my company's current system and I got a little worried about an aspect of the dovecot's design. I was surprised that dovecot doesn't use prefork for its mail processes, forking a new processes for each new client connection.
Login processes are preforked and they can also be configured to keep
reusing existing processes. But yes, post-login imap/pop3 processes
aren't preforked because it would require a larger redesign of the
master process. Which is actually what I have been planning on doing
for v2.0 for a long time now.
But there are even some theoretical problems with preforking. For
example the most secure way to set up your users is to use a different
UNIX UID for each user. So for preforking that means your preforked
processes must run as root until they receive the information about
which UID they need to run as. And the code running as root should be
minimized..
2008/8/14 Timo Sirainen tss@iki.fi:
But there are even some theoretical problems with preforking. For example the most secure way to set up your users is to use a different UNIX UID for each user. So for preforking that means your preforked processes must run as root until they receive the information about which UID they need to run as. And the code running as root should be minimized..
True, but it's a common scenario to have thousands of users, in which case they usually have all the same UID.
[]'s
On Aug 14, 2008, at 4:25 PM, Giorgenes Gelatti wrote:
2008/8/14 Timo Sirainen tss@iki.fi:
But there are even some theoretical problems with preforking. For
example the most secure way to set up your users is to use a different UNIX
UID for each user. So for preforking that means your preforked processes
must run as root until they receive the information about which UID they need
to run as. And the code running as root should be minimized..True, but it's a common scenario to have thousands of users, in which case they usually have all the same UID.
I'm hoping that'll change some day in future when there are better
tools to easily support UID-per-user. :)
Hi Timo,
On 14 Aug 2008, at 17:04, Timo Sirainen wrote:
On Aug 14, 2008, at 2:38 PM, Giorgenes Gelatti wrote:
I've been studying dovecot for replacing my company's current system and I got a little worried about an aspect of the dovecot's design. I was surprised that dovecot doesn't use prefork for its mail processes, forking a new processes for each new client connection.
Login processes are preforked and they can also be configured to
keep reusing existing processes. But yes, post-login imap/pop3
processes aren't preforked because it would require a larger
redesign of the master process. Which is actually what I have been
planning on doing for v2.0 for a long time now.
Do you have pointer(s) to some mails / wiki pages talking about your
ideas of implementation of pre-forked process pool?
Regards, Sebastien Tandel
On Tue, 2008-08-19 at 12:13 -0300, Sebastien Tandel wrote:
Hi Timo,
On 14 Aug 2008, at 17:04, Timo Sirainen wrote:
On Aug 14, 2008, at 2:38 PM, Giorgenes Gelatti wrote:
I've been studying dovecot for replacing my company's current system and I got a little worried about an aspect of the dovecot's design. I was surprised that dovecot doesn't use prefork for its mail processes, forking a new processes for each new client connection.
Login processes are preforked and they can also be configured to
keep reusing existing processes. But yes, post-login imap/pop3
processes aren't preforked because it would require a larger
redesign of the master process. Which is actually what I have been
planning on doing for v2.0 for a long time now.Do you have pointer(s) to some mails / wiki pages talking about your
ideas of implementation of pre-forked process pool?
No. The plans are still pretty vague.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Thu, Aug 14, 2008 at 03:38:50PM -0300, Giorgenes Gelatti wrote:
Hello All,
[...]
It is well known that preforking is a good pratice if you want to achieve a higher performance. When I was asked about it I readily answered: "of course it does". For my surprise later, i doesn't.
With fork latencies in the range of 500 to 1500 microseconds (on Pentium 900 MHz-class hardware!) on most modern kernels[1] I wonder whether this "good practice" isn't on the verge of voodoo ;-)
(Of course, in a http server, where you might expect thousands of connects per second, this is another story -- which is mitigated by HTTP 1.1, when properly streaming several requests per connection).
[1] http://bulk.fefe.de/scalability/, search "The fork benchmark"
Regards
- -- tomás -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux)
iD8DBQFIpR1XBcgs9XrR2kYRAqPdAJ0dbp+fUW0MpWdNvXa3SUvXP3v3eQCcCsTS hFbhMpoG+OjI4i+za6xNn+4= =SRgx -----END PGP SIGNATURE-----
Hi,
It is well known that preforking is a good pratice if you want to achieve a higher performance. When I was asked about it I readily answered: "of course it does".
For my surprise later, i doesn't.With fork latencies in the range of 500 to 1500 microseconds (on
Pentium 900 MHz-class hardware!) on most modern kernels[1] I wonder whether
this "good practice" isn't on the verge of voodoo ;-)
OK, it measures the fork instruction. But fork is using a copy-on-
write mechanism ... It means that *none* of the parent's memory pages
are copied. Each page is simply *shared* by *all* the child /until/ a
modification is made to it.Therefore this test obviously does not
take into account time taken when modifying data. And I strongly
suspect that dovecot is not only doing read-only access to memory
when running. :-/
P.S. : I'm not saying though it is mandatory to have such a mechanism
in dovecot ;)
Regards, Sebastien
(Of course, in a http server, where you might expect thousands of connects per second, this is another story -- which is mitigated by
HTTP 1.1, when properly streaming several requests per connection).
[1] http://bulk.fefe.de/scalability/, search "The fork benchmark"
Regards
- -- tomás -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux)
iD8DBQFIpR1XBcgs9XrR2kYRAqPdAJ0dbp+fUW0MpWdNvXa3SUvXP3v3eQCcCsTS hFbhMpoG+OjI4i+za6xNn+4= =SRgx -----END PGP SIGNATURE-----
The master process exec's the mail process (imap or pop3) after fork.
gpg
2008/8/15 Sebastien Tandel sebastien.tandel@corp.terra.com.br:
Hi,
It is well known that preforking is a good pratice if you want to achieve a higher performance. When I was asked about it I readily answered: "of course it does". For my surprise later, i doesn't.
With fork latencies in the range of 500 to 1500 microseconds (on Pentium 900 MHz-class hardware!) on most modern kernels[1] I wonder whether this "good practice" isn't on the verge of voodoo ;-)
OK, it measures the fork instruction. But fork is using a copy-on-write mechanism ... It means that *none* of the parent's memory pages are copied. Each page is simply *shared* by *all* the child /until/ a modification is made to it.Therefore this test obviously does not take into account time taken when modifying data. And I strongly suspect that dovecot is not only doing read-only access to memory when running. :-/
P.S. : I'm not saying though it is mandatory to have such a mechanism in dovecot ;)
Regards, Sebastien
(Of course, in a http server, where you might expect thousands of connects per second, this is another story -- which is mitigated by HTTP 1.1, when properly streaming several requests per connection).
[1] http://bulk.fefe.de/scalability/, search "The fork benchmark"
Regards
- -- tomás -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux)
iD8DBQFIpR1XBcgs9XrR2kYRAqPdAJ0dbp+fUW0MpWdNvXa3SUvXP3v3eQCcCsTS hFbhMpoG+OjI4i+za6xNn+4= =SRgx -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Fri, Aug 15, 2008 at 03:37:53PM -0300, Sebastien Tandel wrote:
[...]
[fork is fast]
OK, it measures the fork instruction. But fork is using a copy-on-write mechanism ... It means that *none* of the parent's memory pages are copied. Each page is simply *shared* by *all* the child /until/ a modification is made to it.Therefore this test obviously does not take into account time taken when modifying data. And I strongly suspect that dovecot is not only doing read-only access to memory when running. :-/
Yep, but what can you do after pre-forking and before the request comes in?
Thus I'd expect pre-forking not to save us much which can't be saved by prudent programming (save mentioned exception of > thousands of connects per second).
We are now deeply in specula-land, I guess ;-)
Regards
- -- tomás -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux)
iD8DBQFIplsTBcgs9XrR2kYRAlnWAJ9N4no7nCvu9f/psXBpFJdBhYEwMgCZAWfe EPXLY1QCdx999EXv4q/tbf8= =7bCz -----END PGP SIGNATURE-----
On 16 Aug 2008, at 01:44, tomas@tuxteam.de tomas@tuxteam.de wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Fri, Aug 15, 2008 at 03:37:53PM -0300, Sebastien Tandel wrote:
[...]
[fork is fast]
OK, it measures the fork instruction. But fork is using a copy-on- write mechanism ... It means that *none* of the parent's memory pages
are copied. Each page is simply *shared* by *all* the child /until/ a
modification is made to it.Therefore this test obviously does not take into
account time taken when modifying data. And I strongly suspect that dovecot is
not only doing read-only access to memory when running. :-/Yep, but what can you do after pre-forking and before the request
comes in?Thus I'd expect pre-forking not to save us much which can't be
saved by prudent programming (save mentioned exception of > thousands of
connects per second).
I don't really know the dovecot's code but I guess you can't save
much between pre-forking and request comes in. But after the request
comes in, you save a lot if you're re-using the same processes for
several users. Of course, as mentioned by Timo, you should be careful
about security concerns.
We are now deeply in specula-land, I guess ;-)
Yes, we are in specula-land and can't get out of it without having a
reasonable prototype implementation of pre-fork in dovecot. :)
Regards, Sebastien Tandel
participants (5)
-
Giorgenes Gelatti
-
Jose Celestino
-
Sebastien Tandel
-
Timo Sirainen
-
tomas＠tuxteam.de