[Dovecot] Out of memory problem
Hi guys,
Running SuSe 9.3 with dovecot 1.0.alpha3 source installation.
One of the users has quite a lot of important email in a folder, but when he trys to view this folder, it either sits and waits, and then times out, or just logs out straight away.
Looking in the log(/var/log/mail), I'm getting the following error.
Feb 6 09:22:08 mail postfix/qmgr[27221]: 198B96D1CE: removed Feb 6 09:22:08 mail dovecot: imap(ralphs): pool_system_malloc(): Out of memory Feb 6 09:22:08 mail dovecot: child 23018 (imap) returned error 83 (Out of memory)
I've searched around on google for this error and haven't found anything useful as of yet.
My next step is to upgrade to the latest version of dovecot, but this is quite a large company, and I will need to wait until after hours to do the upgrade.
Any help would be greatly appreciated.
Thanks!
Neil Wilson DcData Powered by Linux, driven by passion!
-- This email and all contents are subject to the following disclaimer: http://www.dcdata.co.za/emaildisclaimer.html
On Mon, 2006-02-06 at 11:29 +0200, Neil Wilson wrote:
Hi guys,
Running SuSe 9.3 with dovecot 1.0.alpha3 source installation.
One of the users has quite a lot of important email in a folder, but when he trys to view this folder, it either sits and waits, and then times out, or just logs out straight away.
How many messages does the mailbox have?
Feb 6 09:22:08 mail postfix/qmgr[27221]: 198B96D1CE: removed Feb 6 09:22:08 mail dovecot: imap(ralphs): pool_system_malloc(): Out of memory Feb 6 09:22:08 mail dovecot: child 23018 (imap) returned error 83 (Out of memory)
You could try increasing mail_process_size to larger than the default 256MB. Although that should have been enough for even mailboxes with tens of thousands of messages..
One possibility would also be that the mailbox's index files are broken and Dovecot tries to allocate too much memory, which fails. You could try deleting them and reopening the mailbox. But please make a copy of those files first and if deleting them helps, send them to me so I can try to fix that bug permanently.
dovecot.index and dovecot.index.log files don't contain any sensitive information, but dovecot.index.cache may contain message subjects, from/to/etc. fields, so you might not want to send that without overwriting that data (I could write a program to do that).
Hi Timo,
The user has over 1000 messages in his inbox, and about 150 folders, and I'm not sure how many emails he has in the rest of the folders.
It's only the one folder that isn't responding when I click on it, and it either logs me out straight away when I click on the folder, or sits for about 20 seconds, and then logs me out.
I am using Horde/IMP/ to view the messages, but his email client Thunderbird also does a similar thing.
I tried to delete the index files, and I still get the same error in my mail log.
I see that I am getting the following in my mail.err log.
Feb 7 11:45:38 mail dovecot: imap(ralphs): pool_system_malloc(): Out of memory Feb 7 11:45:38 mail dovecot: child 32473 (imap) returned error 83 (Out of memory) Feb 7 11:47:26 mail dovecot: imap(ralphs): Corrupted index cache file /home/ralphs/Maildir/dovecot.index.cache: Duplicated field in header: hdr.CONTENT-TYPE
This is with the current dovecot.index.cache file, and not the old one that I backed up and then deleted.
Should I send just the index.cache file to you? I don't think overwriting the headers will be necessary.
Thanks.
Neil Wilson
Powered by Linux, driven by passion!
Timo Sirainen wrote:
On Mon, 2006-02-06 at 11:29 +0200, Neil Wilson wrote:
Hi guys,
Running SuSe 9.3 with dovecot 1.0.alpha3 source installation.
One of the users has quite a lot of important email in a folder, but when he trys to view this folder, it either sits and waits, and then times out, or just logs out straight away.
How many messages does the mailbox have?
Feb 6 09:22:08 mail postfix/qmgr[27221]: 198B96D1CE: removed Feb 6 09:22:08 mail dovecot: imap(ralphs): pool_system_malloc(): Out of memory Feb 6 09:22:08 mail dovecot: child 23018 (imap) returned error 83 (Out of memory)
You could try increasing mail_process_size to larger than the default 256MB. Although that should have been enough for even mailboxes with tens of thousands of messages..
One possibility would also be that the mailbox's index files are broken and Dovecot tries to allocate too much memory, which fails. You could try deleting them and reopening the mailbox. But please make a copy of those files first and if deleting them helps, send them to me so I can try to fix that bug permanently.
dovecot.index and dovecot.index.log files don't contain any sensitive information, but dovecot.index.cache may contain message subjects, from/to/etc. fields, so you might not want to send that without overwriting that data (I could write a program to do that).
-- This email and all contents are subject to the following disclaimer: http://www.dcdata.co.za/emaildisclaimer.html
On Tue, 2006-02-07 at 12:00 +0200, Neil Wilson wrote:
Hi Timo,
The user has over 1000 messages in his inbox, and about 150 folders, and I'm not sure how many emails he has in the rest of the folders.
It's only the one folder that isn't responding when I click on it, and it either logs me out straight away when I click on the folder, or sits for about 20 seconds, and then logs me out.
I am using Horde/IMP/ to view the messages, but his email client Thunderbird also does a similar thing.
I tried to delete the index files, and I still get the same error in my mail log.
Well, that's strange then. But since you're using maildir, you should be able to easily check how many mails he has in the mailbox that can't be opened? ie. ls Maildir/.folder.name/cur/|wc -l
Did you try if raising mail_process_size helps?
Feb 7 11:47:26 mail dovecot: imap(ralphs): Corrupted index cache file /home/ralphs/Maildir/dovecot.index.cache: Duplicated field in header: hdr.CONTENT-TYPE
This shouldn't happen, but it looks like there's some bug which causes it to be added twice there. Anyway, this error is harmless and completely transparent to client.
Timo Sirainen wrote:
On Tue, 2006-02-07 at 12:00 +0200, Neil Wilson wrote: Well, that's strange then. But since you're using maildir, you should be able to easily check how many mails he has in the mailbox that can't be opened? ie. ls Maildir/.folder.name/cur/|wc -l
Thanks, I forgot that checking through the client wasn't the only way to see how many mails were in the folders.
Below is the count of mails in the folder.
.Saved\ mails.Legal.Company\ register/cur/ | wc -l 47
Did you try if raising mail_process_size helps? Yes, I raised it to 512MB and it still did the same thing.
Is it worth trying to upgrade to the latest dovecot version, or will I sit with the same problem?
Thanks.
Neil Wilson Powered by Linux, driven by passion!
-- This email and all contents are subject to the following disclaimer: http://www.dcdata.co.za/emaildisclaimer.html
On Wed, 2006-02-08 at 09:24 +0200, Neil Wilson wrote:
Timo Sirainen wrote:
On Tue, 2006-02-07 at 12:00 +0200, Neil Wilson wrote: Well, that's strange then. But since you're using maildir, you should be able to easily check how many mails he has in the mailbox that can't be opened? ie. ls Maildir/.folder.name/cur/|wc -l
Thanks, I forgot that checking through the client wasn't the only way to see how many mails were in the folders.
Below is the count of mails in the folder.
.Saved\ mails.Legal.Company\ register/cur/ | wc -l 47
Did you try if raising mail_process_size helps? Yes, I raised it to 512MB and it still did the same thing.
Is it worth trying to upgrade to the latest dovecot version, or will I sit with the same problem?
A vaguely-related feature request here:
postfix has a really useful 'postconf' command that lets you dump out the postfix configuration (or the non-default set options or the default options) of your current config. the advantage is that it lets you look through all the config options at a glance and look for things related to it.
Could dovecot have something similar? Or maybe does it already have something like this?
thanks -sv
On Wed, 2006-02-08 at 09:24 +0200, Neil Wilson wrote:
Timo Sirainen wrote:
On Tue, 2006-02-07 at 12:00 +0200, Neil Wilson wrote: Well, that's strange then. But since you're using maildir, you should be able to easily check how many mails he has in the mailbox that can't be opened? ie. ls Maildir/.folder.name/cur/|wc -l
Thanks, I forgot that checking through the client wasn't the only way to see how many mails were in the folders.
Below is the count of mails in the folder.
.Saved\ mails.Legal.Company\ register/cur/ | wc -l 47
Did you try if raising mail_process_size helps? Yes, I raised it to 512MB and it still did the same thing.
Is it worth trying to upgrade to the latest dovecot version, or will I sit with the same problem?
I don't remember fixing any important memory leaks, so upgrading probably won't help.
These kinds of problems are a bit difficult to debug. One thing you could do is check with gdb where it dies:
ulimit -v 131072 MAIL=/home/user/Maildir gdb /usr/local/libexec/dovecot/imap b i_fatal_status run
x select "Saved mails.Legal.Company register"
Does it crash here already? If not, do another command:
x fetch 1:* (envelope body rfc822.size internaldate flags)
If it still didn't crash, check with rawlog (see dovecot.org -> bugreports) what Thunderbird is sending and send the same commands.
In any case, when it then finally breaks at i_fatal_status, do "bt full" command and send me the result. Also it would be helpful if the binary still contained debugging information (maybe run it with src/imap/imap instead in Dovecot's source directory where it's at least not stripped).
-- Neil Wilson DcData support@dcdata.co.za support: 0860-1-LINUX http://www.dcdata.co.za Powered by Linux, driven by passion!
Timo Sirainen wrote:
These kinds of problems are a bit difficult to debug. One thing you could do is check with gdb where it dies:
ulimit -v 131072 MAIL=/home/user/Maildir gdb /usr/local/libexec/dovecot/imap b i_fatal_status run
x select "Saved mails.Legal.Company register" Does it crash here already? If not, do another command: Yes it crashed at this step. Below is the bt full output.
(gdb) bt full #0 i_fatal_status (status=83, format=0x46f858 "pool_system_malloc(): Out of memory") at failures.c:187 args = {{gp_offset = 5867792, fp_offset = 0, overflow_arg_area = 0x2aaaaad2d9cf, reg_save_area = 0x2ab2ab0008c0}} #1 0x000000000045dcd0 in pool_system_malloc (pool=<value optimized out>, size=<value optimized out>) at mempool-system.c:73 mem = <value optimized out> #2 0x0000000000459604 in i_malloc (size=<value optimized out>) at imem.c:9 No locals. #3 0x000000000043dec6 in mail_index_keywords_create_from_indexes (t=0x5990e0, keyword_indexes=0x7fffffffdfe0) at mail-index-transaction.c:905 k = <value optimized out> count = <value optimized out> #4 0x000000000041e779 in maildir_sync_index_finish (sync_ctx=0x598820, partial=0) at maildir-sync.c:1049 kw = (struct mail_keywords *) 0x7fffffffdfd0 mbox = (struct maildir_mailbox *) 0x594e20 view = (struct mail_index_view *) 0x598910 iter = (struct maildir_uidlist_iter_ctx *) 0x599380 trans = (struct mail_index_transaction *) 0x5990e0 hdr = (const struct mail_index_header *) 0x598990 rec = <value optimized out> seq = <value optimized out> uid = <value optimized out> uflags = <value optimized out> filename = <value optimized out> flags = <value optimized out> keywords = {buffer = 0x583800, element_size = 4} idx_keywords = {buffer = 0x5838f0, element_size = 4} uid_validity = <value optimized out> next_uid = 0 value = <value optimized out> old_new_sync_time = <value optimized out> ret = 0 full_rescan = 0 #5 0x000000000041eaaa in maildir_sync_context (ctx=0x5833e0, forced=<value optimized out>, sync_last_commit=<value optimized out>) at maildir-sync.c:1268 ret = 0 new_changed = 1 cur_changed = 1 full_rescan = 0 #6 0x000000000041ecbc in maildir_storage_sync_init (box=0x594e20, flags=MAILBOX_SYNC_FLAG_FULL_READ) at maildir-sync.c:1325 mbox = <value optimized out> ---Type <return> to continue, or q <return> to quit--- ctx = (struct maildir_sync_context *) 0x5833e0 ret = 0 #7 0x000000000044a187 in mailbox_sync_init (box=<value optimized out>, flags=<value optimized out>) at mail-storage.c:336 No locals. #8 0x0000000000419500 in imap_sync_nonselected (box=<value optimized out>, flags=<value optimized out>) at imap-sync.c:171 ctx = <value optimized out> sync_rec = {seq1 = 5820128, seq2 = 0, type = 5821688} status = {messages = 0, recent = 0, unseen = 0, uidvalidity = 0, uidnext = 5821568, first_unseen_seq = 0, keywords = 0x58d480} #9 0x000000000041286f in _cmd_select_full (cmd=0x58d4f8, readonly=0) at cmd-select.c:39 client = (struct client *) 0x58d480 storage = (struct mail_storage *) 0x58cee0 box = (struct mailbox *) 0x594e20 status = {messages = 5830872, recent = 0, unseen = 5821688, uidvalidity = 0, uidnext = 5821568, first_unseen_seq = 0, keywords = 0x58bdf0} mailbox = <value optimized out> #10 0x00000000004140b4 in _client_input (context=<value optimized out>) at client.c:334 client = (struct client *) 0x58d480 cmd = (struct client_command_context *) 0x58d4f8 #11 0x000000000045bbd3 in io_loop_handler_run (ioloop=0x58bda0) at ioloop-poll.c:184 ctx = (struct ioloop_handler_context *) 0x58bdf0 pollfd = <value optimized out> tv = {tv_sec = 9, tv_usec = 999314} io = (struct io *) 0x58d780 t_id = 2 msecs = <value optimized out> ret = 0 call = <value optimized out> #12 0x000000000045b12c in io_loop_run (ioloop=0x58bda0) at ioloop.c:230 No locals. #13 0x000000000041b0c6 in main (argc=<value optimized out>, argv=0x7fffffffe2d8, envp=0x7fffffffe2e8) at main.c:229 No locals. (gdb)
still contained debugging information (maybe run it with src/imap/imap instead in Dovecot's source directory where it's at least not stripped). Now run with src/imap/imap
(gdb) bt full #0 i_fatal_status (status=83, format=0x46f858 "pool_system_malloc(): Out of memory") at failures.c:187 args = {{gp_offset = 5867792, fp_offset = 0, overflow_arg_area = 0x2aaaaad2d9cf, reg_save_area = 0x2ab2ab0008c0}} #1 0x000000000045dcd0 in pool_system_malloc (pool=<value optimized out>, size=<value optimized out>) at mempool-system.c:73 mem = <value optimized out> #2 0x0000000000459604 in i_malloc (size=<value optimized out>) at imem.c:9 No locals. #3 0x000000000043dec6 in mail_index_keywords_create_from_indexes (t=0x5990e0, keyword_indexes=0x7fffffffdfd0) at mail-index-transaction.c:905 k = <value optimized out> count = <value optimized out> #4 0x000000000041e779 in maildir_sync_index_finish (sync_ctx=0x598820, partial=0) at maildir-sync.c:1049 kw = (struct mail_keywords *) 0x7fffffffdfc0 mbox = (struct maildir_mailbox *) 0x594e20 view = (struct mail_index_view *) 0x598910 iter = (struct maildir_uidlist_iter_ctx *) 0x599380 trans = (struct mail_index_transaction *) 0x5990e0 hdr = (const struct mail_index_header *) 0x598990 rec = <value optimized out> seq = <value optimized out> uid = <value optimized out> uflags = <value optimized out> filename = <value optimized out> flags = <value optimized out> keywords = {buffer = 0x583800, element_size = 4} idx_keywords = {buffer = 0x5838f0, element_size = 4} uid_validity = <value optimized out> next_uid = 0 value = <value optimized out> old_new_sync_time = <value optimized out> ret = 0 full_rescan = 0 #5 0x000000000041eaaa in maildir_sync_context (ctx=0x5833e0, forced=<value optimized out>, sync_last_commit=<value optimized out>) at maildir-sync.c:1268 ret = 0 new_changed = 1 cur_changed = 1 full_rescan = 0 #6 0x000000000041ecbc in maildir_storage_sync_init (box=0x594e20, flags=MAILBOX_SYNC_FLAG_FULL_READ) at maildir-sync.c:1325 mbox = <value optimized out> ---Type <return> to continue, or q <return> to quit--- ctx = (struct maildir_sync_context *) 0x5833e0 ret = 0 #7 0x000000000044a187 in mailbox_sync_init (box=<value optimized out>, flags=<value optimized out>) at mail-storage.c:336 No locals. #8 0x0000000000419500 in imap_sync_nonselected (box=<value optimized out>, flags=<value optimized out>) at imap-sync.c:171 ctx = <value optimized out> sync_rec = {seq1 = 5820128, seq2 = 0, type = 5821688} status = {messages = 0, recent = 0, unseen = 0, uidvalidity = 0, uidnext = 5821568, first_unseen_seq = 0, keywords = 0x58d480} #9 0x000000000041286f in _cmd_select_full (cmd=0x58d4f8, readonly=0) at cmd-select.c:39 client = (struct client *) 0x58d480 storage = (struct mail_storage *) 0x58cee0 box = (struct mailbox *) 0x594e20 status = {messages = 5830872, recent = 0, unseen = 5821688, uidvalidity = 0, uidnext = 5821568, first_unseen_seq = 0, keywords = 0x58bdf0} mailbox = <value optimized out> #10 0x00000000004140b4 in _client_input (context=<value optimized out>) at client.c:334 client = (struct client *) 0x58d480 cmd = (struct client_command_context *) 0x58d4f8 #11 0x000000000045bbd3 in io_loop_handler_run (ioloop=0x58bda0) at ioloop-poll.c:184 ctx = (struct ioloop_handler_context *) 0x58bdf0 pollfd = <value optimized out> tv = {tv_sec = 9, tv_usec = 999173} io = (struct io *) 0x58d780 t_id = 2 msecs = <value optimized out> ret = 0 call = <value optimized out> #12 0x000000000045b12c in io_loop_run (ioloop=0x58bda0) at ioloop.c:230 No locals. #13 0x000000000041b0c6 in main (argc=<value optimized out>, argv=0x7fffffffe2c8, envp=0x7fffffffe2d8) at main.c:229 No locals. (gdb)
-- This email and all contents are subject to the following disclaimer: http://www.dcdata.co.za/emaildisclaimer.html
On Wed, 2006-02-08 at 10:28 +0200, Neil Wilson wrote:
#3 0x000000000043dec6 in mail_index_keywords_create_from_indexes (t=0x5990e0, keyword_indexes=0x7fffffffdfe0) at mail-index-transaction.c:905 k = <value optimized out> count = <value optimized out>
Oh, you have a 64bit system. This bug has actually been fixed in 1.0betas already. :)
Timo Sirainen wrote:
Oh, you have a 64bit system. This bug has actually been fixed in 1.0betas already. :)
Ah ok cool.
Downloaded dovecot-1.0.beta2 ran './configure' with no problems or errors, then ran 'make' I get the following message.
gcc -std=gnu99 -g -O2 -Wall -W -Wmissing-prototypes
-Wmissing-declarations -Wpointer-arith -Wchar-subscripts -Wformat=2
-Wbad-function-cast -o dovecot auth-process.o askpass.o dict-process.o
log.o login-process.o mail-process.o main.o master-settings.o
syslog-util.o ssl-init.o ssl-init-gnutls.o ssl-init-openssl.o
../lib-settings/libsettings.a ../lib/liblib.a
main.o(.text+0x1233): In function main': /usr/src/dovecot-1.0.beta2/src/master/main.c:778: undefined reference to ssl_check_parameters_file'
collect2: ld returned 1 exit status
make[3]: *** [dovecot] Error 1
make[3]: Leaving directory /usr/src/dovecot-1.0.beta2/src/master' make[2]: *** [all-recursive] Error 1 make[2]: Leaving directory /usr/src/dovecot-1.0.beta2/src'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/usr/src/dovecot-1.0.beta2'
make: *** [all] Error 2
To me it looks like a problem with SSL, but by default my ./configure isn't compiling with SSL support.
Install prefix ...................... : /usr/local File offsets ........................ : 64bit I/O loop method ..................... : poll File change notification method ..... : dnotify Building with SSL support ........... : no Building with IPv6 support .......... : yes Building with pop3 server ........... : yes Building with GSSAPI support ........ : no Building with user database modules . : static prefetch passwd passwd-file checkpassword (modules) Building with password lookup modules : passwd passwd-file shadow checkpassword (modules)
Any ideas?
Thanks.
Regards.
Neil Wilson
-- This email and all contents are subject to the following disclaimer: http://www.dcdata.co.za/emaildisclaimer.html
Timo Sirainen wrote:
Oh, you have a 64bit system. This bug has actually been fixed in 1.0betas already. :)
Thanks for all your help Timo.
It's now all working!
Regards.
Neil Wilson Powered by Linux, driven by passion!
-- This email and all contents are subject to the following disclaimer: http://www.dcdata.co.za/emaildisclaimer.html
participants (3)
-
Neil Wilson
-
seth vidal
-
Timo Sirainen