Re: [Dovecot] imap memory footprint rather large
[back to list]
On Tue, 2007-08-14 at 14:50 +0300, Timo Sirainen wrote:
On Tue, 2007-08-14 at 13:47 +0200, martin f krafft wrote:
also sprach Timo Sirainen tss@iki.fi [2007.08.14.1250 +0200]:
mmap()ing a 200MB file doesn't mean that it immediately uses 200MB of memory. It just reserves that much virtual space. Even if it happens to read the whole 200MB, it shouldn't be much different from simply reading the file in small parts, in which case kernel places the same 200MB to page cache.
I am aware of that. I just wonder why dovecot has to read the entire file into memory, via read() or mmap().
It doesn't. It mmaps it entirely so it doesn't waste CPU mmaping and unmapping parts of it. It's then accessed only when and where needed.
It probably does access the whole file though. This is because all the cached data for messages are stored next to each others. So now that your client fetches only INTERNALDATE it doesn't really need to read much, but because mmaped pages are read 4kB at a time, it practically reads the whole file.
Perhaps this could use some work in future too, so that fields that are accessed together are stored in cache file close together. But then if multiple clients are used, there can be conflicting access patterns, and what's good for one client would be bad for another. Data could also be duplicated to reduce disk reads for different access patterns, but that comes at the cost of disk space..
also sprach Timo Sirainen tss@iki.fi [2007.08.14.1358 +0200]:
So I guess most of the data in your dovecot.index.cache file came from some initial FETCH ENVELOPE/BODYSTRUCTURE/etc. for all messages. If you delete it, it won't probably get as large anymore.
This is true, I deleted it and it went back to 12 Mb, taking most of our performance problems with it.
I'm not sure if there's anything I can do on Dovecot's side to make this work better. This shouldn't be a problem except for large mailboxes that are accessed with Dovecot for the first time. There the possibilities are to cache wanted data immediately so that it can be accessed fast the next time, or not cache it at all the first time and if it's needed again doing the whole thing all over again.
Well, that, and dovecot could expire data in the cache after a while, right?
-- martin; (greetings from the heart of the sun.) \____ echo mailto: !#^."<*>"|tr "<*> mailto:" net@madduck
a gourmet concerned about calories is like a punter eyeing the clock.
spamtraps: madduck.bogus@madduck.net
also sprach martin f krafft madduck@madduck.net [2007.08.14.1552 +0200]:
So I guess most of the data in your dovecot.index.cache file came from some initial FETCH ENVELOPE/BODYSTRUCTURE/etc. for all messages. If you delete it, it won't probably get as large anymore.
This is true, I deleted it and it went back to 12 Mb, taking most of our performance problems with it.
I found the file to be ever growing, so when it had grown back to 160Mb in a single day, I decided to employ cron on the mail server:
11 4 * * * find $HOME/.maildir -type f -name dovecot.index.cache -exec rm {} \;
Since my mail is fetched in the background anyway, I am happy to take the performance hit first thing in the morning.
-- martin; (greetings from the heart of the sun.) \____ echo mailto: !#^."<*>"|tr "<*> mailto:" net@madduck
"moderation is a fatal thing. enough is as bad as a meal. more than enough is as good as a feast." -- oscar wilde
spamtraps: madduck.bogus@madduck.net
I found the file to be ever growing, so when it had grown back to 160Mb in a single day, I decided to employ cron on the mail server:
11 4 * * * find $HOME/.maildir -type f -name dovecot.index.cache -exec rm {} \;
Since my mail is fetched in the background anyway, I am happy to take the performance hit first thing in the morning.
I'm wondering, can this be done safely with connected users?
-- Leroy
Leroy van Logchem wrote:
I found the file to be ever growing, so when it had grown back to 160Mb in a single day, I decided to employ cron on the mail server:
11 4 * * * find $HOME/.maildir -type f -name dovecot.index.cache -exec rm {} \;
Since my mail is fetched in the background anyway, I am happy to take the performance hit first thing in the morning.
I'm wondering, can this be done safely with connected users?
My guess would be : Yes.
If dovecot has the file open, the OS won't trash it until there are no more links.
If dovecot is about to open the file, I'm sure Timo would use an open/create-on-fail pattern, not a test-if-exist/open patter which would lead to a race.
Worst case I can see is someone's session suddenly slows down a bit.
-- Curtis Maloney cmaloney@cardgate.net
participants (4)
-
Curtis Maloney
-
Leroy van Logchem
-
martin f krafft
-
Timo Sirainen