[Dovecot] dovecot memory consumption beta vs. old "stable" series
Hi!
I just wanted to know if anyone can give a definitive answer if dovecot beta's memory requirements are much better than in the old pre-alpha "stable" series?
I'm using a rather old stable.20050712 release on an also rather old Debian Sarge box to manage my personal mails. I'm pretty satisfied with this dovecot release but it's memory consumption when deleting messages causes major troubles on this box - more than 500 MB of virtual memory allocated to the "imap" process is quite a lot and often causes random processes to fall victim to the OOM killer. :-(
Adding even more swap space also is no viable option. (The system has 64 MB RAM and currently about 650 MB swap space.)
So, would it be worth a try to install dovecot beta2 or something when it's released, or won't it help in this regard? I won't want to sacrificy the otherwise well working installation, if dovecot beta2 isn't guaranteed to help I'll continue to monitor available memory and swap space while cleaning up my spams...
Greetings,
Gunter
-- +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ they were directly in front of him. -- Mort goes out for a drink
- http://aachen.uni-dsl.de/ - Der direkte Draht in's Hochschulnetz! + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ "What do people like to drink here, then?" The landlord looked sideways at his customers, a clever trick given that
(Terry Pratchett, Mort)+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+PGP-verschlüsselte Mails bevorzugt! +
On Fri, 2006-01-20 at 14:01 +0100, Gunter Ohrner wrote:
I just wanted to know if anyone can give a definitive answer if dovecot beta's memory requirements are much better than in the old pre-alpha "stable" series?
Probably somewhat similar.
I'm using a rather old stable.20050712 release on an also rather old Debian Sarge box to manage my personal mails. I'm pretty satisfied with this dovecot release but it's memory consumption when deleting messages causes major troubles on this box - more than 500 MB of virtual memory allocated to the "imap" process is quite a lot and often causes random processes to fall victim to the OOM killer. :-(
Umm. If it's allocating 500MB when deleting messages there's something seriously wrong. I haven't heard this happening with any Dovecot version before.
With "deleting" do you mean marking messages as deleted, or the expunge operation? With mbox or maildir? Does it matter how many messages you're deleting? How large the mailbox itself is?
Whoops, sorry for the late answer, I somehow missed your response!
Am Sonntag, 22. Januar 2006 13:33, schrieb Timo Sirainen:
dovecot beta's memory requirements are much better than in the old Probably somewhat similar.
Mh, ok.
messages causes major troubles on this box - more than 500 MB of virtual memory allocated to the "imap" process is quite a lot and often causes random processes to fall victim to the OOM killer. :-( Umm. If it's allocating 500MB when deleting messages there's something seriously wrong. I haven't heard this happening with any Dovecot version before.
Ok. Well, it does and it's easily reproduceable with this stable release. :-)
I want to upgrade dovecot to a recent beta. If the problem also happens with this version, I could try to help to track it down. Or should I first do some tests with the currently installed release?
With "deleting" do you mean marking messages as deleted, or the expunge operation?
It seems to be the marking. It happens if I "delete" messages from kMail, instead of moving them to trash. I'm not sure what happens under the hood if I do, but the message files seem to stay there, at least until the imap process exits cleanly.
With mbox or maildir?
maildir
Does it matter how many messages you're deleting?
Yes, that's the reason. I run into it if I clean up my Spam which I do in about 2 week intervals. I just tested it, deleting (marking) 6000 to 7000 files caused dovecot to grow to more than 200 MB:
9253 gunter 9 0 203m 14m 12m S 0.0 23.0 6:40.52 imap
I noticed that the REAL memory consumption starts if dovecot gets OOM killed in the first try and I retry after artificially increasing the swap space size. If repeating the same operation after this kill, dovecot easily eats twice as much VIRT memory than it would have before. (This "would have" is extrapolated from experience in this case, of course.)
Maybe it's somehow index related, if the killed process leaves stale or corrupt index files?
How large the mailbox itself is?
Currently about 17000 files. (My Spam since the beginning of January.)
Thanks,
Gunter
-- *** Powered by AudioScrobbler --> http://www.last.fm/user/Interneci/ *** 00:00 | Lullacry - Don't Touch the Flame 23:57 | Paradise Lost - Self Obsessed 23:52 | Therion - Secret of the Runes 23:47 | Within Temptation - See Who I Am *** PGP-Verschlüsselung bei eMails erwünscht :-) *** PGP: 0x1128F25F ***
On Tue, 2006-02-14 at 00:53 +0100, Gunter Ohrner wrote:
messages causes major troubles on this box - more than 500 MB of virtual memory allocated to the "imap" process is quite a lot and often causes random processes to fall victim to the OOM killer. :-( Umm. If it's allocating 500MB when deleting messages there's something seriously wrong. I haven't heard this happening with any Dovecot version before.
Ok. Well, it does and it's easily reproduceable with this stable release. :-)
I want to upgrade dovecot to a recent beta. If the problem also happens with this version, I could try to help to track it down. Or should I first do some tests with the currently installed release?
Probably easier if you just try if beta3 has fixed this already, so we don't waste time for nothing :)
Hi Timo!
Am Dienstag, 14. Februar 2006 09:33, schrieb Timo Sirainen:
messages causes major troubles on this box - more than 500 MB of virtual memory allocated to the "imap" process is quite a lot and often causes random processes to fall victim to the OOM killer. Probably easier if you just try if beta3 has fixed this already, so we don't waste time for nothing :)
Ok, I just installed and tested beta3. The occassional "hang" of the connection I experienced with the ancient "stable" release seems to be gone now.
However, deleting (marking as deleted) large batches of messages still requires lots of RAM:
28507 gunter 17 0 211m 30m 8572 R 82.9 50.5 6:45.00 imap
After the operation finishes, dovecot remains at a VIRT size of 209m.
This look pretty much for me, even for deleting 7000 messages in a row.
This machine has 64m of physical RAM and runs the 2.4.27 Linux kernel from Debian Sarge.
I've not tried yet if the memory utilization additionally doubles on the second try if dovecot got OOM killed on the first - I added sufficient swap space before the test - but I could do if this would be interesting and/or helpful for you.
Greetings,
Gunter
-- +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Programming today is a race between software engineers striving to build bigger and better idiot-proof programs, and the Universe trying to produce bigger and better idiots. So far, the Universe is winning. +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+PGP-verschlüsselte Mails bevorzugt! +
On Wed, 2006-02-15 at 00:42 +0100, Gunter Ohrner wrote:
However, deleting (marking as deleted) large batches of messages still requires lots of RAM:
28507 gunter 17 0 211m 30m 8572 R 82.9 50.5 6:45.00 imap
After the operation finishes, dovecot remains at a VIRT size of 209m.
This look pretty much for me, even for deleting 7000 messages in a row.
I just tried testing this myself in a mailbox with around 10000 mails. I tried deleting everything, I tried deleting messages from here and there (sorted the mails based on size and deleted half of them), etc. At no point did the process's size grow more than a few hundred kilobytes..
So, how large is the mailbox you're deleting messages from (ie. how many messages do you leave there)? With mbox or maildir? What client are you using to delete them? Are you deleting the messages in sequential order?
Am Montag, 20. Februar 2006 18:44, schrieb Timo Sirainen:
I just tried testing this myself in a mailbox with around 10000 mails. I tried deleting everything, I tried deleting messages from here and there (sorted the mails based on size and deleted half of them), etc. At no point did the process's size grow more than a few hundred kilobytes..
That's strange... In this case I really like to know what I'm doing differently...
My imap process *always* needs several megabytes, though I do not know how much of it is mmaped indexes - it'll probably the majority of the memory consumption displayed.
So, how large is the mailbox you're deleting messages from (ie. how many messages do you leave there)?
IIRC: For example 17000 messages before, deleting 7000 in a row, leaving 10000 after the imap process has shut down.
With mbox or maildir?
maildir
What client are you using to delete them?
kMail
Are you deleting the messages in sequential order?
Probably not, I sort by date in kmail, select a block covering about two weeks of spams and hit "delete".
Can dovecot log the commands issued by the client? I read something about a "rawlog"? As I'm currently using ssl only, etheral is useless as-is and I'd have to reconfigure dovecot.
I'd like to log all IMAP commands and try a few things tomorrow (The server machine is an old box and takes a while to process larger requests, especially if they cause the imap process to consume 3 to 4 times more virtual memory than the box physically has... ;), collecting /proc/PID/maps etc.
Greetings,
Gunter
-- *** Powered by AudioScrobbler --> http://www.last.fm/user/Interneci/ *** 01:05 | Traci Lords - Control (Juno Reactor Instrumental) 00:56 | Juno Reactor - Navras 00:51 | Juno Reactor - Mona Lisa Overdrive 00:45 | Juno Reactor - Zwara *** PGP-Verschlüsselung bei eMails erwünscht :-) *** PGP: 0x1128F25F ***
On Tue, 2006-02-21 at 01:09 +0100, Gunter Ohrner wrote:
I'd like to log all IMAP commands and try a few things tomorrow (The server machine is an old box and takes a while to process larger requests, especially if they cause the imap process to consume 3 to 4 times more virtual memory than the box physically has... ;), collecting /proc/PID/maps etc.
If you have a recent kernel the smaps file will be of interest too because it tells you what is shared, memory mapped etc.
johannes
participants (3)
-
Gunter Ohrner
-
Johannes Berg
-
Timo Sirainen