replicator crashing - oom

[Daniel Miller]

Sorry for the re-post - just want to make sure you saw this:

#0  0x00007fddaf597c37 in __GI_raise (sig=sig at entry=6) at 
../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007fddaf59b028 in __GI_abort () at abort.c:89
#2  0x00007fddaf9c0c86 in default_fatal_finish (type=<optimized out>, 
status=status at entry=0) at failures.c:201
#3  0x00007fddaf9c0d6e in i_internal_fatal_handler (ctx=0x7fff7197d000, 
format=<optimized out>, args=<optimized out>)
     at failures.c:670
#4  0x00007fddaf958322 in i_panic (format=format at entry=0x7fddafa047b8 
"data stack: Out of memory when allocating %lu bytes")
     at failures.c:275
#5  0x00007fddaf9bee3f in mem_block_alloc (min_size=536870912) at 
data-stack.c:341
#6  0x00007fddaf9bf08b in t_malloc_real (size=size at entry=536870912, 
permanent=permanent at entry=true) at data-stack.c:396
#7  0x00007fddaf9bf0ca in t_malloc (size=size at entry=536870912) at 
data-stack.c:442
#8  0x00007fddaf9da5f8 in pool_data_stack_realloc (pool=<optimized out>, 
mem=0x7fdd99f59038, old_size=268435449, new_size=536870912)
     at mempool-datastack.c:126
#9  0x00007fddaf9bcc42 in p_realloc (new_size=<optimized out>, 
old_size=<optimized out>, mem=<optimized out>, pool=<optimized out>)
     at mempool.h:88
#10 buffer_alloc (buf=buf at entry=0xf849a8, size=536870912) at buffer.c:36
#11 0x00007fddaf9bcfb4 in buffer_check_limits (data_size=32, 
pos=268435424, buf=0xf849a8) at buffer.c:75
#12 buffer_write (_buf=0xf849a8, pos=268435424, data=0xfa0420, 
data_size=32) at buffer.c:187
#13 0x000000000040519b in array_append_i (count=1, data=0xfa0420, 
array=<synthetic pointer>) at ../../../src/lib/array.h:168
#14 replicator_queue_handle_sync_lookups (user=0xfa3fc0, queue=0xf92260) 
at replicator-queue.c:278
---Type <return> to continue, or q <return> to quit---
#15 replicator_queue_push (queue=0xf92260, user=0xfa3fc0) at 
replicator-queue.c:296
#16 0x00000000004049c2 in dsync_callback (reply=DSYNC_REPLY_OK,
     state=0xf84590 "AQAAAHX+sC3O3gNSjAoAAOEwx0RQoclMAwAAAAQ", 'A' 
<repeats 20 times>, "DAAAAEu2cNfSIZk0oMgAAO8QcX0yhyUwDAAAAlg", 'A' 
<repeats 20 times>, "IAAADFjqMEUPIEU+IlAACdtazLUaHJTAcAAAAI", 'A' 
<repeats 20 times>, "BwAAAEfYPTqXct9VYRMAAJ21"..., context=0xfa1560)
     at replicator-brain.c:121
#17 0x00000000004040a8 in dsync_callback (client=0xfa14a0, 
state=<optimized out>, reply=DSYNC_REPLY_OK) at dsync-client.c:65
#18 0x0000000000404275 in dsync_input_line (line=0xf84988 "+", 
client=0xfa14a0) at dsync-client.c:132
#19 dsync_input (client=0xfa14a0) at dsync-client.c:153
#20 0x00007fddaf9d47d2 in io_loop_call_io (io=0xfa15a0) at ioloop.c:599
#21 0x00007fddaf9d5d17 in io_loop_handler_run_internal 
(ioloop=ioloop at entry=0xf8c720) at ioloop-epoll.c:223
#22 0x00007fddaf9d486c in io_loop_handler_run 
(ioloop=ioloop at entry=0xf8c720) at ioloop.c:648
#23 0x00007fddaf9d4a28 in io_loop_run (ioloop=0xf8c720) at ioloop.c:623
#24 0x00007fddaf961fc3 in master_service_run (service=0xf8c5c0, 
callback=callback at entry=0x404720 <client_connected>)
     at master-service.c:641
#25 0x000000000040342a in main (argc=1, argv=0xf8c390) at replicator.c:112

Daniel

On 3/23/2017 12:17 AM, Aki Tuomi wrote:
> sysctl kernel.core_pattern usually indicates where cores are placed. If
> it says 'core' you are probably not gonna find it.
>
> Aki
>
>
> On 22.03.2017 18:11, Daniel Miller wrote:
>> Where would I find the core file?  I'm not finding anything obvious.
>>
>> The replicator path is /usr/local/libexec/dovecot/replicator
>>
>> Daniel
>>
>> On 3/22/2017 12:52 AM, Aki Tuomi wrote:
>>> Can you provide us gdb bt full dump?
>>>
>>> gdb /usr/libexec/dovecot/replicator /path/to/core
>>>
>>> on some systems, it's /usr/lib/dovecot/replicator
>>>
>>> Aki
>>>
>>> On 21.03.2017 23:48, Daniel Miller wrote:
>>>> I have the following in my log:
>>>>
>>>> Mar 21 14:46:59 bubba dovecot: replicator: Panic: data stack: Out of
>>>> memory when allocating 1073741864 bytes
>>>> Mar 21 14:46:59 bubba dovecot: replicator: Error: Raw backtrace:
>>>> /usr/local/lib/dovecot/libdovecot.so.0(+0x97c90) [0x7f4638a7cc90] ->
>>>> /usr/local/lib/dovecot/libdovecot.so.0(+0x97d6e) [0x7f4638a7cd6e] ->
>>>> /usr/local/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7f4638a14322] ->
>>>> /usr/local/lib/dovecot/libdovecot.so.0(+0x95e3f) [0x7f4638a7ae3f] ->
>>>> /usr/local/lib/dovecot/libdovecot.so.0(+0x9608b) [0x7f4638a7b08b] ->
>>>> /usr/local/lib/dovecot/libdovecot.so.0(+0xb15f8) [0x7f4638a965f8] ->
>>>> /usr/local/lib/dovecot/libdovecot.so.0(+0x93c42) [0x7f4638a78c42] ->
>>>> /usr/local/lib/dovecot/libdovecot.so.0(buffer_write+0x74)
>>>> [0x7f4638a78fb4] -> dovecot/replicator(replicator_queue_push+0x13b)
>>>> [0x40519b] -> dovecot/replicator() [0x4049c2] -> dovecot/replicator()
>>>> [0x4040a8] -> dovecot/replicator() [0x404275] ->
>>>> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x52)
>>>> [0x7f4638a907d2] ->
>>>> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0xe7)
>>>>
>>>> [0x7f4638a91d17] ->
>>>> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x3c)
>>>> [0x7f4638a9086c] ->
>>>> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_run+0x38)
>>>> [0x7f4638a90a28] ->
>>>> /usr/local/lib/dovecot/libdovecot.so.0(master_service_run+0x13)
>>>> [0x7f4638a1dfc3] -> dovecot/replicator(main+0x17a) [0x40342a] ->
>>>> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)
>>>> [0x7f4638640ec5] -> dovecot/replicator() [0x4034c5]
>>>> Mar 21 14:47:00 bubba dovecot: replicator: Fatal: master:
>>>> service(replicator): child 15650 killed with signal 6 (core dumped)
>>>>