[Dovecot] imap memory footprint rather large

[Timo Sirainen]

[back to list]

On Tue, 2007-08-14 at 14:50 +0300, Timo Sirainen wrote:
> On Tue, 2007-08-14 at 13:47 +0200, martin f krafft wrote:
> > also sprach Timo Sirainen <tss at iki.fi> [2007.08.14.1250 +0200]:
> > > mmap()ing a 200MB file doesn't mean that it immediately uses 200MB
> > > of memory. It just reserves that much virtual space. Even if it
> > > happens to read the whole 200MB, it shouldn't be much different
> > > from simply reading the file in small parts, in which case kernel
> > > places the same 200MB to page cache.
> > 
> > I am aware of that. I just wonder why dovecot has to read the entire
> > file into memory, via read() or mmap().
> 
> It doesn't. It mmaps it entirely so it doesn't waste CPU mmaping and
> unmapping parts of it. It's then accessed only when and where needed.

It probably does access the whole file though. This is because all the
cached data for messages are stored next to each others. So now that
your client fetches only INTERNALDATE it doesn't really need to read
much, but because mmaped pages are read 4kB at a time, it practically
reads the whole file.

Perhaps this could use some work in future too, so that fields that are
accessed together are stored in cache file close together. But then if
multiple clients are used, there can be conflicting access patterns, and
what's good for one client would be bad for another. Data could also be
duplicated to reduce disk reads for different access patterns, but that
comes at the cost of disk space..

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url : http://dovecot.org/pipermail/dovecot/attachments/20070814/1cc17284/attachment.bin