On Fri 16 Feb 2007 at 12:41PM, Timo Sirainen wrote:
On Fri, 2007-02-16 at 01:06 -0800, Dan Price wrote:
More data: watching another user suffer from this and watching just the mmap's, I see:
vvvvvvvvmmap64(0x00000000, 16973824, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANON... mmap64(0x00000000, 16982016, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANON,... mmap64(0x00000000, 16990208, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANON,... mmap64(0x00000000, 16998400, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANON,... .... mmap64(0x00000000, 20602880, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANON,... mmap64(0x00000000, 20611072, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANON,... mmap64(0x00000000, 20619264, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANON,... ... mmap64(0x00000000, 25067520, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANON... ...
So this sure looks serious-- it's almost like a memory leak.
How large is the dovecot.index.cache file for that user? The anonymous mmaps are created so that Dovecot can "mmap" the dovecot.index.cache file into memory without actually using mmap().
The munmapping happens because Dovecot wants to grow the mmaped area, and since Solaris doesn't have mremap() call it fakes it by doing another larger mmap() and moving the data in 1MB blocks while unmapping the old memory at the same time.
So.. Hmm. Could the problem simply be that the mmap-copy-growing is too slow? If the user really has some 25MB cache file, that could be it. I think I could change the code so that it mmap()s immediately enough memory to fit the whole cache file. Probably a good idea to do anyway.
Cool-- of course, I missed that we were growing at pagesize (8K).
So that sounds plausible. I can file an RFE for mremap, as well-- frankly this is the first time I've heard of it. The user has a large mailbox: 600+MB. Here is the contents of their .imap/INBOX directory:
drwx------ 2 comay staff 6 Feb 16 01:00 ./ drwx------ 3 comay staff 3 Feb 15 12:03 ../ -rw------- 1 comay staff 2.0M Feb 16 01:00 dovecot.index -rw------- 1 comay staff 48M Feb 16 00:16 dovecot.index.cache -rw------- 1 comay staff 844 Feb 16 01:00 dovecot.index.log -rw------- 1 comay staff 434K Feb 16 00:27 dovecot.index.log.2
I agree that mmaping the whole index.cache in one shot is the way to go-- asking the VM system to do work in little chunks is (at least on Solaris) a real performance killer.
-dp-- Daniel Price - Solaris Kernel Engineering - dp@eng.sun.com - blogs.sun.com/dp