[Dovecot] severe performance problem (mail cache related?)

[Dan Price]

On Fri 16 Feb 2007 at 12:41PM, Timo Sirainen wrote:
> On Fri, 2007-02-16 at 01:06 -0800, Dan Price wrote:
> > More data: watching another user suffer from this and watching just the mmap's,
> > I see:
> > 
> >                    vvvvvvvv
> > mmap64(0x00000000, 16973824, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANON...
> > mmap64(0x00000000, 16982016, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANON,...
> > mmap64(0x00000000, 16990208, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANON,...
> > mmap64(0x00000000, 16998400, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANON,...
> > ....
> > mmap64(0x00000000, 20602880, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANON,...
> > mmap64(0x00000000, 20611072, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANON,...
> > mmap64(0x00000000, 20619264, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANON,...
> > ...
> > mmap64(0x00000000, 25067520, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANON...
> > ...
> > 
> > So this sure looks serious-- it's almost like a memory leak.
> 
> How large is the dovecot.index.cache file for that user? The anonymous
> mmaps are created so that Dovecot can "mmap" the dovecot.index.cache
> file into memory without actually using mmap().
> 
> The munmapping happens because Dovecot wants to grow the mmaped area,
> and since Solaris doesn't have mremap() call it fakes it by doing
> another larger mmap() and moving the data in 1MB blocks while unmapping
> the old memory at the same time.
>
> So.. Hmm. Could the problem simply be that the mmap-copy-growing is too
> slow? If the user really has some 25MB cache file, that could be it. I
> think I could change the code so that it mmap()s immediately enough
> memory to fit the whole cache file. Probably a good idea to do anyway.

Cool-- of course, I missed that we were growing at pagesize (8K).

So that sounds plausible.  I can file an RFE for mremap, as well--
frankly this is the first time I've heard of it.  The user has a large
mailbox: 600+MB.  Here is the contents of their .imap/INBOX directory:

drwx------   2 comay    staff          6 Feb 16 01:00 ./
drwx------   3 comay    staff          3 Feb 15 12:03 ../
-rw-------   1 comay    staff       2.0M Feb 16 01:00 dovecot.index
-rw-------   1 comay    staff        48M Feb 16 00:16 dovecot.index.cache
-rw-------   1 comay    staff        844 Feb 16 01:00 dovecot.index.log
-rw-------   1 comay    staff       434K Feb 16 00:27 dovecot.index.log.2

I agree that mmaping the whole index.cache in one shot is the way to
go-- asking the VM system to do work in little chunks is (at least
on Solaris) a real performance killer.

        -dp

-- 
Daniel Price - Solaris Kernel Engineering - dp at eng.sun.com - blogs.sun.com/dp