Jakob Hirsch put forth on 3/16/2010 3:55 AM:
Stan Hoeppner, 2010-03-16 08:45:
Regarding memory usage, on Linux anyway, you should really read this Timo. It would be mutually beneficial to Dovecot (and Postfix, and etc). I've not played with it yet, and I don't have a busy server to test it on, but I'm betting it would seriously benefit those running servers with hundreds of concurrent users. I'm going to spin this into my next kernel and play with it.
AFAIK, KSM is primarily useful for vservers (xen, kvm etc.) running the same software. How would running a single instance of dovecot (or any other daemon) profit by that? AFAICS, "generates many instances of the same data" does not apply to dovecot (or most other deamons).
You didn't read the paste Jakob. Or maybe you don't comprehend the material? Also, it seems you're unaware that Dovecot creates at minimum 1 imap process per client connection. In the case of Thunderbird, it creates 5 imap processes. Each one is over 2MB in size, and all of those imap processes share close to 2MB of memory pages that are identical, because the code is identical. If they're all identical, why have 500 copies in memory instead of one?
KSM was designed during the KVM effort, but it's applicable to any program duplication in memory, and Dovecot is a perfect example of an application which would benefit from KSM due to the large number of identical processes. On a "loaded" Dovecot server you will have hundreds of imap, and possibly dovecot-auth, processes--depends on whether login_process_per_connection = yes. If yes, you'll have one dovecot-auth per client. It appears from my vantage point as a non programmer that each of these processes has about 2MB of duplicate memory. So on a server with 500 concurrent users, assuming 1 imap connection for each (although most MUAs open more than one connection per session--TB opens 5), KSM can potentially save you ~2GB-6GB of memory by de-duplicating the identical page contents of these Dovecot processes.
In the case of TB, which I have quite a bit of experience with, 1 imap process ends up servicing all the requests, while the other 4 sit idle, just eating memory senselessly. KSM would eliminate most of that memory waste. Personally I've changed my TB to use a single connection instead of 5, and there is zero difference in performance. Too bad the team sets the default at 5 instead of 1.
Fast forward to a threaded Dovecot design, and you still benefit from KSM, because all the imap worker threads are duplicate code, probably the same for the auth workers. The benefit won't be as great as with the 1 imap process per connection design of today as some of the code will only be duplicated 2 x nCPUs times, the master processes, whereas right now all the code is duplicated across all imap worker processes.
In summary, if KSM is configured correctly, and Dovecot coded to work with KSM, the memory savings could be huge on large servers.
-- Stan