[Dovecot] "Out of memory" error in 1.0alpha3
Hi All,
I have Dovecot 1.0 Alpha3 running on x86_64 Linux 2.6 multi-CPU box using Maildir folders. Moving messages from one folder to another in Thunderbird 1.0.7, regardless of their size (usually small), produced error 83, "Out of memory", when both of these parameters were unset (default): mail_read_mmaped, mmap_disable. The log line before the error was pool_system_malloc(): Out of memory. I initially changed mmap_disable to "yes", the error stopped, but it still wasn't moving messages reliably. I then set MM parameters more aggressively: mail_read_mmaped = yes, mmap_disable = no (and the third of the series, mmap_no_write, is left unset). The error stopped popping up, and the moving works speedily. Maybe the default for mail_read_mmaped should be "yes?" Thank you!
Sergey.
On Fri, 2005-10-14 at 00:47 -0400, Sergey A. Lipnevich wrote:
Hi All,
I have Dovecot 1.0 Alpha3 running on x86_64 Linux 2.6 multi-CPU box using Maildir folders. Moving messages from one folder to another in Thunderbird 1.0.7, regardless of their size (usually small), produced error 83, "Out of memory", when both of these parameters were unset (default): mail_read_mmaped, mmap_disable. The log line before the error was pool_system_malloc(): Out of memory.
This is a bug in Dovecot. It looks like it happens only with 64bit systems. Would you mind getting a gdb backtrace from that, it would help in fixing it? Works basically like:
- Open the mailbox in thunderbird
- gdb attach pid-of-the-imap-process-that's-going-to-move-the-messages
- Give "cont" command
- Move the messages and wait for crash
- Give "bt" command
- mail the reply to me
I initially changed mmap_disable to "yes", the error stopped, but it still wasn't moving messages reliably. I then set MM parameters more aggressively: mail_read_mmaped = yes, mmap_disable = no (and the third of the series, mmap_no_write, is left unset). The error stopped popping up, and the moving works speedily. Maybe the default for mail_read_mmaped should be "yes?"
That probably just goes around the buggy code path..
Timo,
I'll try to reproduce the error on a different AMD64 machine (it won't have multiple CPUs though). The one I was talking about is in production. BTW, the compiler was GCC 4.0.2. Dovecot is great. Thanks a lot!
Sergey.
Timo Sirainen wrote:
On Fri, 2005-10-14 at 00:47 -0400, Sergey A. Lipnevich wrote:
Hi All,
I have Dovecot 1.0 Alpha3 running on x86_64 Linux 2.6 multi-CPU box using Maildir folders. Moving messages from one folder to another in Thunderbird 1.0.7, regardless of their size (usually small), produced error 83, "Out of memory", when both of these parameters were unset (default): mail_read_mmaped, mmap_disable. The log line before the error was pool_system_malloc(): Out of memory.
This is a bug in Dovecot. It looks like it happens only with 64bit systems. Would you mind getting a gdb backtrace from that, it would help in fixing it? Works basically like:
- Open the mailbox in thunderbird
- gdb attach pid-of-the-imap-process-that's-going-to-move-the-messages
- Give "cont" command
- Move the messages and wait for crash
- Give "bt" command
- mail the reply to me
I initially changed mmap_disable to "yes", the error stopped, but it still wasn't moving messages reliably. I then set MM parameters more aggressively: mail_read_mmaped = yes, mmap_disable = no (and the third of the series, mmap_no_write, is left unset). The error stopped popping up, and the moving works speedily. Maybe the default for mail_read_mmaped should be "yes?"
That probably just goes around the buggy code path..
Timo,
In regards to the conversation below, does this item in release notes for 1.0.alpha4 mean that you found the problem?
- epoll was broken with 64bit systems
With 1.0.alpha3, I'm still seeing an "Out of memory" error with one Apple Mail user (but the user says everything works just fine), so we're planning to upgrade to 1.0.alpha4 and take it from there. Thank you! Sergey.
Timo Sirainen wrote:
On Fri, 2005-10-14 at 00:47 -0400, Sergey A. Lipnevich wrote:
Hi All,
I have Dovecot 1.0 Alpha3 running on x86_64 Linux 2.6 multi-CPU box using Maildir folders. Moving messages from one folder to another in Thunderbird 1.0.7, regardless of their size (usually small), produced error 83, "Out of memory", when both of these parameters were unset (default): mail_read_mmaped, mmap_disable. The log line before the error was pool_system_malloc(): Out of memory.
This is a bug in Dovecot. It looks like it happens only with 64bit systems. Would you mind getting a gdb backtrace from that, it would help in fixing it? Works basically like:
- Open the mailbox in thunderbird
- gdb attach pid-of-the-imap-process-that's-going-to-move-the-messages
- Give "cont" command
- Move the messages and wait for crash
- Give "bt" command
- mail the reply to me
On Sat, 2005-10-22 at 18:14 -0400, Sergey A. Lipnevich wrote:
Timo,
In regards to the conversation below, does this item in release notes for 1.0.alpha4 mean that you found the problem?
- epoll was broken with 64bit systems
With 1.0.alpha3, I'm still seeing an "Out of memory" error with one Apple Mail user (but the user says everything works just fine), so we're planning to upgrade to 1.0.alpha4 and take it from there.
Well, I'm not sure. I suppose it could have happened to you because of it, if you really had been using epoll. If it still happens, a gdb backtrace really would be helpful in fixing it :)
Hi all,
On 2005-10-27, Timo Sirainen tss@iki.fi wrote:
[1
] On Sat, 2005-10-22 at 18:14 -0400, Sergey A. Lipnevich wrote: With 1.0.alpha3, I'm still seeing an "Out of memory" error with one Apple Mail user (but the user says everything works just fine), so we're planning to upgrade to 1.0.alpha4 and take it from there.
Well, I'm not sure. I suppose it could have happened to you because of it, if you really had been using epoll. If it still happens, a gdb backtrace really would be helpful in fixing it :)
I have a backtrace on 1.0.alpha4, amd64, 1 CPU, dovecot compiled with gcc 4.0. I was getting the error with Wanderlust, getting the status of some (not all) mailboxes.
Breakpoint 1, i_fatal_status (status=83, format=0x472bc0 "pool_system_malloc(): Out of memory") at failures.c:187 187 { (gdb) bt #0 i_fatal_status (status=83, format=0x472bc0 "pool_system_malloc(): Out of memory") at failures.c:187 #1 0x000000000046053e in pool_system_malloc (pool=Variable "pool" is not available. ) at mempool-system.c:73 #2 0x000000000043ecb6 in mail_index_keywords_create_from_indexes (t=0x599660, keyword_indexes=0x7fbffff7a0) at mail-index-transaction.c:905 #3 0x000000000041f1e7 in maildir_sync_index_finish (sync_ctx=0x597db0, partial=0) at maildir-sync.c:1049 #4 0x000000000041f6dd in maildir_sync_context (ctx=0x586358, forced=Variable "forced" is not available. ) at maildir-sync.c:1268 #5 0x000000000041f933 in maildir_storage_sync_init (box=0x59b680, flags=0) at maildir-sync.c:1325 #6 0x000000000041a000 in imap_sync_nonselected (box=Variable "box" is not available. ) at imap-sync.c:177 #7 0x0000000000413022 in cmd_status (cmd=0x590468) at cmd-status.c:68 #8 0x0000000000414319 in _client_input (context=Variable "context" is not available. ) at client.c:338 #9 0x000000000045e4a6 in io_loop_handler_run (ioloop=0x58ec70) at ioloop-poll.c:190 #10 0x000000000045d9dd in io_loop_run (ioloop=0x58ec70) at ioloop.c:230 #11 0x000000000041b841 in main (argc=Variable "argc" is not available. ) at main.c:232
Oh, and one more thing: The dying process leaves a .lock file behind, dovecot-uidlist.lock. It causes all later processes to loop endlessly, trying to acquire the lock. I thought processes should clean up after themselves when they croak. (-:
Thanks,
Andreas Fuchs, asf@boinkor.net, asf@jabber.at, antifuchs
On 13.11.2005, at 13:56, Andreas Fuchs wrote:
#2 0x000000000043ecb6 in mail_index_keywords_create_from_indexes (t=0x599660, keyword_indexes=0x7fbffff7a0) at mail-index-transaction.c:905
Oh. This was pretty much the same problem as the previous x86-64 OOM that I fixed, only the last one was for mail_index_keywords_create() function, 10 lines above this one.. Fixed in CVS.
On 2005-12-02, Timo Sirainen tss@iki.fi wrote:
[1
] On 13.11.2005, at 13:56, Andreas Fuchs wrote: #2 0x000000000043ecb6 in mail_index_keywords_create_from_indexes (t=0x599660, keyword_indexes=0x7fbffff7a0) at mail-index-transaction.c:905
Oh. This was pretty much the same problem as the previous x86-64 OOM that I fixed, only the last one was for mail_index_keywords_create() function, 10 lines above this one.. Fixed in CVS.
I extracted the patch from CVS and applied it to my alpha4 tree. It seems to work now (at least there are no crashes on the mailboxes that didn't work before). Thanks a lot!
-- Andreas Fuchs, asf@boinkor.net, asf@jabber.at, antifuchs
Hi all,
On 2005-10-27, Timo Sirainen tss@iki.fi wrote:
[1
] On Sat, 2005-10-22 at 18:14 -0400, Sergey A. Lipnevich wrote: With 1.0.alpha3, I'm still seeing an "Out of memory" error with one Apple Mail user (but the user says everything works just fine), so we're planning to upgrade to 1.0.alpha4 and take it from there.
Well, I'm not sure. I suppose it could have happened to you because of it, if you really had been using epoll. If it still happens, a gdb backtrace really would be helpful in fixing it :)
I have a backtrace on 1.0.alpha4, amd64, 1 CPU, dovecot compiled with gcc 4.0. I was getting the error with Wanderlust, getting the status of some (not all) mailboxes.
Breakpoint 1, i_fatal_status (status=83, format=0x472bc0 "pool_system_malloc(): Out of memory") at failures.c:187 187 { (gdb) bt #0 i_fatal_status (status=83, format=0x472bc0 "pool_system_malloc(): Out of memory") at failures.c:187 #1 0x000000000046053e in pool_system_malloc (pool=Variable "pool" is not available. ) at mempool-system.c:73 #2 0x000000000043ecb6 in mail_index_keywords_create_from_indexes (t=0x599660, keyword_indexes=0x7fbffff7a0) at mail-index-transaction.c:905 #3 0x000000000041f1e7 in maildir_sync_index_finish (sync_ctx=0x597db0, partial=0) at maildir-sync.c:1049 #4 0x000000000041f6dd in maildir_sync_context (ctx=0x586358, forced=Variable "forced" is not available. ) at maildir-sync.c:1268 #5 0x000000000041f933 in maildir_storage_sync_init (box=0x59b680, flags=0) at maildir-sync.c:1325 #6 0x000000000041a000 in imap_sync_nonselected (box=Variable "box" is not available. ) at imap-sync.c:177 #7 0x0000000000413022 in cmd_status (cmd=0x590468) at cmd-status.c:68 #8 0x0000000000414319 in _client_input (context=Variable "context" is not available. ) at client.c:338 #9 0x000000000045e4a6 in io_loop_handler_run (ioloop=0x58ec70) at ioloop-poll.c:190 #10 0x000000000045d9dd in io_loop_run (ioloop=0x58ec70) at ioloop.c:230 #11 0x000000000041b841 in main (argc=Variable "argc" is not available. ) at main.c:232
Oh, and one more thing: The dying process leaves a .lock file behind, dovecot-uidlist.lock. It causes all later processes to loop endlessly, trying to acquire the lock. I thought processes should clean up after themselves when they croak. (-:
Thanks,
Andreas Fuchs, asf@boinkor.net, asf@jabber.at, antifuchs
Sergey A. Lipnevich a écrit :
Timo,
In regards to the conversation below, does this item in release notes for 1.0.alpha4 mean that you found the problem?
- epoll was broken with 64bit systems
With 1.0.alpha3, I'm still seeing an "Out of memory" error with one Apple Mail user (but the user says everything works just fine), so we're planning to upgrade to 1.0.alpha4 and take it from there.
I'm using alpha4 on x86_64 Linux and still seeing this precise error.
-- Nico
Same here on alpha4 on x86_64! This is somehow x86_64 related I think!
BR, Alen
Nicolas STRANSKY wrote:
Sergey A. Lipnevich a écrit :
Timo,
In regards to the conversation below, does this item in release notes for 1.0.alpha4 mean that you found the problem?
- epoll was broken with 64bit systems
With 1.0.alpha3, I'm still seeing an "Out of memory" error with one Apple Mail user (but the user says everything works just fine), so we're planning to upgrade to 1.0.alpha4 and take it from there.
I'm using alpha4 on x86_64 Linux and still seeing this precise error.
participants (5)
-
Alen Salamun
-
Andreas Fuchs
-
Nicolas STRANSKY
-
Sergey A. Lipnevich
-
Timo Sirainen