On Fri 16 Feb 2007 at 12:20AM, Dan Price wrote:
So here we see dovecot allocate a whopping 12MB of anonymous memory, then munmap it all away in 1MB chunks.
Timo, do you know why this is happening? It is ruining performance for me.
More data: watching another user suffer from this and watching just the mmap's, I see:
vvvvvvvvmmap64(0x00000000, 16973824, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANON... mmap64(0x00000000, 16982016, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANON,... mmap64(0x00000000, 16990208, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANON,... mmap64(0x00000000, 16998400, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANON,... .... mmap64(0x00000000, 20602880, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANON,... mmap64(0x00000000, 20611072, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANON,... mmap64(0x00000000, 20619264, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANON,... ... mmap64(0x00000000, 25067520, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANON... ...
So this sure looks serious-- it's almost like a memory leak.
Using DTrace I was able to pinpoint the stack track of the mmap system call:
$ dtrace -n 'syscall::mmap64:entry/pid==164794/{ustack()}'
12 52690 mmap64:entry
libc.so.1mmap64+0xc imapmremap_anon+0xb8
imapfile_cache_set_size+0xe0 imapfile_cache_read+0xa4
imapmail_cache_map+0xec imapmail_cache_get_record+0x20
imapmail_cache_foreach_rec+0x10 imapmail_cache_foreach+0xb4
imapmail_cache_field_exists+0x84 imapmail_cache_lookup_field+0xc
imapindex_mail_get_fixed_field+0x4c imapindex_mail_set_seq+0xbc
imapmail_set_seq+0xc imapindex_storage_search_next+0x70
imapmailbox_search_next+0x14 imapcmd_search+0xf4
imapclient_handle_input+0x138 imapclient_handle_input+0xb0
imap_client_input+0x68 imapio_loop_handler_run+0x140
I also have a complete trace of the flow of control between the return from one call to mmap and the start of the next, but it's 165K, so I have not attached it.
The user reported to me that pine was "doing a selection" or "doing a search" when this happened.
-dp-- Daniel Price - Solaris Kernel Engineering - dp@eng.sun.com - blogs.sun.com/dp