Panic: data stack: Out of memory when allocating bytes

[Thomas Robers]

Hi,

Am 23.01.2018 um 20:07 schrieb Josef 'Jeff' Sipek:
> On Tue, Jan 23, 2018 at 14:03:27 -0500, Josef 'Jeff' Sipek wrote:
>> On Tue, Jan 23, 2018 at 18:21:38 +0100, Thomas Robers wrote:
>>> Hello,
>>>
>>> I'm using Dovecot 2.3 and sometimes i get this:
>>>
>>> --- snip ---
>>> Jan 23 14:23:13 mail dovecot: imap(bob at tutech.de)<4880><PDqibHFjMvrAqG1n>:
>>> Panic: data stack: Out of memory when allocating 134217768 bytes
>>
>> Interesting... imap is trying to allocate 128MB and failing.  A couple of
>> questions:
>>
>> 0. Does this user have any unusually large emails?

No, not usually but there are some mails which are larger than 15mb.
But that's not the normal size. Most e-mail are between some kb up to
5mb.


>> 1. Do you have any idea what the imap process was doing at the time of the
>>     allocation failure?

Yes perhaps. We use shared mailboxes and at the time of failure the
imap process is reading acl files in a shared mailbox (and subfolder).
This shared mailbox has about 2800 subfolder and the acl files are read
in about 10sec and and during this reading the imap process dies with
the already mentioned error. It seems it has something to do with the
shared mailbox, because since yesterday morning 5 core dumps have been
created, 4 of them by one user accessing the shared mailbox and 1
by the user who is the owner of the shared mailbox. No other mailboxes
are affected until now.

>> 2. You snipped all the important parts of the back trace. :)  It *starts* on
>>     the line:
>> 	#0  0x00007f73f1386495 in raise () from /lib64/libc.so.6
> 
> In case you haven't used gdb before...  after starting up gdb, run "bt full"
> at the gdb prompt.  That'll print out a very detailed backtrace.  (You might
> want to sanity check it to make sure there aren't any user passwords in it
> before posting it here...)

Yes, sorry I'm not very familiar in using gdb, but here is the full
backtrace:

--- snip ---
(gdb) bt full
#0  0x00007f73f1386495 in raise () from /lib64/libc.so.6
No symbol table info available.
#1  0x00007f73f1387c75 in abort () from /lib64/libc.so.6
No symbol table info available.
#2  0x00007f73f17ab822 in ?? () from /usr/lib64/dovecot/libdovecot.so.0
No symbol table info available.
#3  0x00007f73f17abc18 in ?? () from /usr/lib64/dovecot/libdovecot.so.0
No symbol table info available.
#4  0x00007f73f17abdeb in t_malloc0 () from 
/usr/lib64/dovecot/libdovecot.so.0
No symbol table info available.
#5  0x00007f73f17a95dd in ?? () from /usr/lib64/dovecot/libdovecot.so.0
No symbol table info available.
#6  0x00007f73f17a967b in buffer_create_dynamic () from 
/usr/lib64/dovecot/libdovecot.so.0
No symbol table info available.
#7  0x00007f73f17a803b in backtrace_get () from 
/usr/lib64/dovecot/libdovecot.so.0
No symbol table info available.
#8  0x00007f73f17af1da in ?? () from /usr/lib64/dovecot/libdovecot.so.0
No symbol table info available.
#9  0x00007f73f17af766 in ?? () from /usr/lib64/dovecot/libdovecot.so.0
No symbol table info available.
#10 0x00007f73f1723e11 in i_panic () from /usr/lib64/dovecot/libdovecot.so.0
No symbol table info available.
#11 0x00007f73f17ab83a in ?? () from /usr/lib64/dovecot/libdovecot.so.0
No symbol table info available.
#12 0x00007f73f17abc18 in ?? () from /usr/lib64/dovecot/libdovecot.so.0
No symbol table info available.
#13 0x00007f73f17abd3b in t_buffer_get () from 
/usr/lib64/dovecot/libdovecot.so.0
No symbol table info available.
#14 0x00007f73f17e1d60 in vstrconcat () from 
/usr/lib64/dovecot/libdovecot.so.0
No symbol table info available.
#15 0x00007f73f17b7d03 in i_strconcat () from 
/usr/lib64/dovecot/libdovecot.so.0
No symbol table info available.
#16 0x00007f73f0b1fa39 in ?? () from /usr/lib64/dovecot/lib01_acl_plugin.so
No symbol table info available.
#17 0x00007f73f0b20740 in ?? () from /usr/lib64/dovecot/lib01_acl_plugin.so
No symbol table info available.
#18 0x00007f73f0b20b7d in acl_backend_vfile_acllist_rebuild () from 
/usr/lib64/dovecot/lib01_acl_plugin.so
No symbol table info available.
#19 0x00007f73f0b21569 in acl_backend_vfile_object_update () from 
/usr/lib64/dovecot/lib01_acl_plugin.so
No symbol table info available.
#20 0x00007f73f0b24bd8 in ?? () from /usr/lib64/dovecot/lib01_acl_plugin.so
No symbol table info available.
#21 0x00007f73f1aa1973 in mailbox_create () from 
/usr/lib64/dovecot/libdovecot-storage.so.0
No symbol table info available.
#22 0x00007f73f1fd1654 in cmd_create ()
No symbol table info available.
#23 0x00007f73f1fde585 in command_exec ()
No symbol table info available.
#24 0x00007f73f1fdb7b0 in ?? ()
No symbol table info available.
#25 0x00007f73f1fdb848 in ?? ()
No symbol table info available.
#26 0x00007f73f1fdbc35 in client_handle_input ()
No symbol table info available.
#27 0x00007f73f1fdc17e in client_input ()
No symbol table info available.
#28 0x00007f73f17c5ec5 in io_loop_call_io () from 
/usr/lib64/dovecot/libdovecot.so.0
No symbol table info available.
#29 0x00007f73f17c7dcf in io_loop_handler_run_internal () from 
/usr/lib64/dovecot/libdovecot.so.0
No symbol table info available.
#30 0x00007f73f17c5fb5 in io_loop_handler_run () from 
/usr/lib64/dovecot/libdovecot.so.0
No symbol table info available.
#31 0x00007f73f17c61d8 in io_loop_run () from 
/usr/lib64/dovecot/libdovecot.so.0
No symbol table info available.
#32 0x00007f73f1745ab3 in master_service_run () from 
/usr/lib64/dovecot/libdovecot.so.0
No symbol table info available.
#33 0x00007f73f1feacee in main ()
No symbol table info available.
-- snip ---

>>     Having the backtrace should help answer question number 1.
>> 3. How big is this process when it dies?

I don't know which imap process dies beforehand so how do i get this 
information?

>> 4. Do you have any sort of ulimit/rlimit set on dovecot in whatever startup
>>     script you use?

The startup script sets no ulimits and ulimit -a shows this

--- snip ---
ulimit -a
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 31380
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 8192
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 31380
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
--- snip ---


>> 5. Do you use cgroups to limit memory usage?

No.

>> 6. Did you disable memory overcommit on the system?

No, i didn't disable it.

-- snip ---
cat /proc/sys/vm/overcommit_memory
0
--- snip ---


>> Jeff.

Thomas.