[back to list]
On Tue, 2007-08-14 at 14:50 +0300, Timo Sirainen wrote:
On Tue, 2007-08-14 at 13:47 +0200, martin f krafft wrote:
also sprach Timo Sirainen tss@iki.fi [2007.08.14.1250 +0200]:
mmap()ing a 200MB file doesn't mean that it immediately uses 200MB of memory. It just reserves that much virtual space. Even if it happens to read the whole 200MB, it shouldn't be much different from simply reading the file in small parts, in which case kernel places the same 200MB to page cache.
I am aware of that. I just wonder why dovecot has to read the entire file into memory, via read() or mmap().
It doesn't. It mmaps it entirely so it doesn't waste CPU mmaping and unmapping parts of it. It's then accessed only when and where needed.
It probably does access the whole file though. This is because all the cached data for messages are stored next to each others. So now that your client fetches only INTERNALDATE it doesn't really need to read much, but because mmaped pages are read 4kB at a time, it practically reads the whole file.
Perhaps this could use some work in future too, so that fields that are accessed together are stored in cache file close together. But then if multiple clients are used, there can be conflicting access patterns, and what's good for one client would be bad for another. Data could also be duplicated to reduce disk reads for different access patterns, but that comes at the cost of disk space..