I have the following in my log:
Mar 21 14:46:59 bubba dovecot: replicator: Panic: data stack: Out of memory when allocating 1073741864 bytes Mar 21 14:46:59 bubba dovecot: replicator: Error: Raw backtrace: /usr/local/lib/dovecot/libdovecot.so.0(+0x97c90) [0x7f4638a7cc90] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x97d6e) [0x7f4638a7cd6e] -> /usr/local/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7f4638a14322] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x95e3f) [0x7f4638a7ae3f] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x9608b) [0x7f4638a7b08b] -> /usr/local/lib/dovecot/libdovecot.so.0(+0xb15f8) [0x7f4638a965f8] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x93c42) [0x7f4638a78c42] -> /usr/local/lib/dovecot/libdovecot.so.0(buffer_write+0x74) [0x7f4638a78fb4] -> dovecot/replicator(replicator_queue_push+0x13b) [0x40519b] -> dovecot/replicator() [0x4049c2] -> dovecot/replicator() [0x4040a8] -> dovecot/replicator() [0x404275] -> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x52) [0x7f4638a907d2] -> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0xe7) [0x7f4638a91d17] -> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x3c) [0x7f4638a9086c] -> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7f4638a90a28] -> /usr/local/lib/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7f4638a1dfc3] -> dovecot/replicator(main+0x17a) [0x40342a] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f4638640ec5] -> dovecot/replicator() [0x4034c5] Mar 21 14:47:00 bubba dovecot: replicator: Fatal: master: service(replicator): child 15650 killed with signal 6 (core dumped)
-- Daniel
Can you provide us gdb bt full dump?
gdb /usr/libexec/dovecot/replicator /path/to/core
on some systems, it's /usr/lib/dovecot/replicator
Aki
On 21.03.2017 23:48, Daniel Miller wrote:
I have the following in my log:
Mar 21 14:46:59 bubba dovecot: replicator: Panic: data stack: Out of memory when allocating 1073741864 bytes Mar 21 14:46:59 bubba dovecot: replicator: Error: Raw backtrace: /usr/local/lib/dovecot/libdovecot.so.0(+0x97c90) [0x7f4638a7cc90] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x97d6e) [0x7f4638a7cd6e] -> /usr/local/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7f4638a14322] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x95e3f) [0x7f4638a7ae3f] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x9608b) [0x7f4638a7b08b] -> /usr/local/lib/dovecot/libdovecot.so.0(+0xb15f8) [0x7f4638a965f8] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x93c42) [0x7f4638a78c42] -> /usr/local/lib/dovecot/libdovecot.so.0(buffer_write+0x74) [0x7f4638a78fb4] -> dovecot/replicator(replicator_queue_push+0x13b) [0x40519b] -> dovecot/replicator() [0x4049c2] -> dovecot/replicator() [0x4040a8] -> dovecot/replicator() [0x404275] -> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x52) [0x7f4638a907d2] -> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0xe7) [0x7f4638a91d17] -> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x3c) [0x7f4638a9086c] -> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7f4638a90a28] -> /usr/local/lib/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7f4638a1dfc3] -> dovecot/replicator(main+0x17a) [0x40342a] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f4638640ec5] -> dovecot/replicator() [0x4034c5] Mar 21 14:47:00 bubba dovecot: replicator: Fatal: master: service(replicator): child 15650 killed with signal 6 (core dumped)
Where would I find the core file? I'm not finding anything obvious.
The replicator path is /usr/local/libexec/dovecot/replicator
Daniel
On 3/22/2017 12:52 AM, Aki Tuomi wrote:
Can you provide us gdb bt full dump?
gdb /usr/libexec/dovecot/replicator /path/to/core
on some systems, it's /usr/lib/dovecot/replicator
Aki
On 21.03.2017 23:48, Daniel Miller wrote:
I have the following in my log:
Mar 21 14:46:59 bubba dovecot: replicator: Panic: data stack: Out of memory when allocating 1073741864 bytes Mar 21 14:46:59 bubba dovecot: replicator: Error: Raw backtrace: /usr/local/lib/dovecot/libdovecot.so.0(+0x97c90) [0x7f4638a7cc90] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x97d6e) [0x7f4638a7cd6e] -> /usr/local/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7f4638a14322] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x95e3f) [0x7f4638a7ae3f] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x9608b) [0x7f4638a7b08b] -> /usr/local/lib/dovecot/libdovecot.so.0(+0xb15f8) [0x7f4638a965f8] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x93c42) [0x7f4638a78c42] -> /usr/local/lib/dovecot/libdovecot.so.0(buffer_write+0x74) [0x7f4638a78fb4] -> dovecot/replicator(replicator_queue_push+0x13b) [0x40519b] -> dovecot/replicator() [0x4049c2] -> dovecot/replicator() [0x4040a8] -> dovecot/replicator() [0x404275] -> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x52) [0x7f4638a907d2] -> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0xe7) [0x7f4638a91d17] -> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x3c) [0x7f4638a9086c] -> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7f4638a90a28] -> /usr/local/lib/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7f4638a1dfc3] -> dovecot/replicator(main+0x17a) [0x40342a] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f4638640ec5] -> dovecot/replicator() [0x4034c5] Mar 21 14:47:00 bubba dovecot: replicator: Fatal: master: service(replicator): child 15650 killed with signal 6 (core dumped)
sysctl kernel.core_pattern usually indicates where cores are placed. If it says 'core' you are probably not gonna find it.
Aki
On 22.03.2017 18:11, Daniel Miller wrote:
Where would I find the core file? I'm not finding anything obvious.
The replicator path is /usr/local/libexec/dovecot/replicator
Daniel
On 3/22/2017 12:52 AM, Aki Tuomi wrote:
Can you provide us gdb bt full dump?
gdb /usr/libexec/dovecot/replicator /path/to/core
on some systems, it's /usr/lib/dovecot/replicator
Aki
On 21.03.2017 23:48, Daniel Miller wrote:
I have the following in my log:
Mar 21 14:46:59 bubba dovecot: replicator: Panic: data stack: Out of memory when allocating 1073741864 bytes Mar 21 14:46:59 bubba dovecot: replicator: Error: Raw backtrace: /usr/local/lib/dovecot/libdovecot.so.0(+0x97c90) [0x7f4638a7cc90] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x97d6e) [0x7f4638a7cd6e] -> /usr/local/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7f4638a14322] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x95e3f) [0x7f4638a7ae3f] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x9608b) [0x7f4638a7b08b] -> /usr/local/lib/dovecot/libdovecot.so.0(+0xb15f8) [0x7f4638a965f8] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x93c42) [0x7f4638a78c42] -> /usr/local/lib/dovecot/libdovecot.so.0(buffer_write+0x74) [0x7f4638a78fb4] -> dovecot/replicator(replicator_queue_push+0x13b) [0x40519b] -> dovecot/replicator() [0x4049c2] -> dovecot/replicator() [0x4040a8] -> dovecot/replicator() [0x404275] -> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x52) [0x7f4638a907d2] -> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0xe7)
[0x7f4638a91d17] -> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x3c) [0x7f4638a9086c] -> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7f4638a90a28] -> /usr/local/lib/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7f4638a1dfc3] -> dovecot/replicator(main+0x17a) [0x40342a] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f4638640ec5] -> dovecot/replicator() [0x4034c5] Mar 21 14:47:00 bubba dovecot: replicator: Fatal: master: service(replicator): child 15650 killed with signal 6 (core dumped)
Sorry for the re-post - just want to make sure you saw this:
#0 0x00007fddaf597c37 in __GI_raise (sig=sig@entry=6) at
../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007fddaf59b028 in __GI_abort () at abort.c:89
#2 0x00007fddaf9c0c86 in default_fatal_finish (type=<optimized out>,
status=status@entry=0) at failures.c:201
#3 0x00007fddaf9c0d6e in i_internal_fatal_handler (ctx=0x7fff7197d000,
format=<optimized out>, args=<optimized out>)
at failures.c:670
#4 0x00007fddaf958322 in i_panic (format=format@entry=0x7fddafa047b8
"data stack: Out of memory when allocating %lu bytes")
at failures.c:275
#5 0x00007fddaf9bee3f in mem_block_alloc (min_size=536870912) at
data-stack.c:341
#6 0x00007fddaf9bf08b in t_malloc_real (size=size@entry=536870912,
permanent=permanent@entry=true) at data-stack.c:396
#7 0x00007fddaf9bf0ca in t_malloc (size=size@entry=536870912) at
data-stack.c:442
#8 0x00007fddaf9da5f8 in pool_data_stack_realloc (pool=<optimized out>,
mem=0x7fdd99f59038, old_size=268435449, new_size=536870912)
at mempool-datastack.c:126
#9 0x00007fddaf9bcc42 in p_realloc (new_size=<optimized out>,
old_size=<optimized out>, mem=<optimized out>, pool=<optimized out>)
at mempool.h:88
#10 buffer_alloc (buf=buf@entry=0xf849a8, size=536870912) at buffer.c:36
#11 0x00007fddaf9bcfb4 in buffer_check_limits (data_size=32,
pos=268435424, buf=0xf849a8) at buffer.c:75
#12 buffer_write (_buf=0xf849a8, pos=268435424, data=0xfa0420,
data_size=32) at buffer.c:187
#13 0x000000000040519b in array_append_i (count=1, data=0xfa0420,
array=<synthetic pointer>) at ../../../src/lib/array.h:168
#14 replicator_queue_handle_sync_lookups (user=0xfa3fc0, queue=0xf92260)
at replicator-queue.c:278
---Type <return> to continue, or q <return> to quit---
#15 replicator_queue_push (queue=0xf92260, user=0xfa3fc0) at
replicator-queue.c:296
#16 0x00000000004049c2 in dsync_callback (reply=DSYNC_REPLY_OK,
state=0xf84590 "AQAAAHX+sC3O3gNSjAoAAOEwx0RQoclMAwAAAAQ", 'A'
Daniel
On 3/23/2017 12:17 AM, Aki Tuomi wrote:
sysctl kernel.core_pattern usually indicates where cores are placed. If it says 'core' you are probably not gonna find it.
Aki
On 22.03.2017 18:11, Daniel Miller wrote:
Where would I find the core file? I'm not finding anything obvious.
The replicator path is /usr/local/libexec/dovecot/replicator
Daniel
On 3/22/2017 12:52 AM, Aki Tuomi wrote:
Can you provide us gdb bt full dump?
gdb /usr/libexec/dovecot/replicator /path/to/core
on some systems, it's /usr/lib/dovecot/replicator
Aki
On 21.03.2017 23:48, Daniel Miller wrote:
I have the following in my log:
Mar 21 14:46:59 bubba dovecot: replicator: Panic: data stack: Out of memory when allocating 1073741864 bytes Mar 21 14:46:59 bubba dovecot: replicator: Error: Raw backtrace: /usr/local/lib/dovecot/libdovecot.so.0(+0x97c90) [0x7f4638a7cc90] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x97d6e) [0x7f4638a7cd6e] -> /usr/local/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7f4638a14322] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x95e3f) [0x7f4638a7ae3f] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x9608b) [0x7f4638a7b08b] -> /usr/local/lib/dovecot/libdovecot.so.0(+0xb15f8) [0x7f4638a965f8] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x93c42) [0x7f4638a78c42] -> /usr/local/lib/dovecot/libdovecot.so.0(buffer_write+0x74) [0x7f4638a78fb4] -> dovecot/replicator(replicator_queue_push+0x13b) [0x40519b] -> dovecot/replicator() [0x4049c2] -> dovecot/replicator() [0x4040a8] -> dovecot/replicator() [0x404275] -> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x52) [0x7f4638a907d2] -> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0xe7)
[0x7f4638a91d17] -> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x3c) [0x7f4638a9086c] -> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7f4638a90a28] -> /usr/local/lib/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7f4638a1dfc3] -> dovecot/replicator(main+0x17a) [0x40342a] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f4638640ec5] -> dovecot/replicator() [0x4034c5] Mar 21 14:47:00 bubba dovecot: replicator: Fatal: master: service(replicator): child 15650 killed with signal 6 (core dumped)
On 24 Mar 2017, at 3.46, Daniel Miller dmiller@amfes.com wrote:
#14 replicator_queue_handle_sync_lookups (user=0xfa3fc0, queue=0xf92260) at replicator-queue.c:278
Oh, you're using synchronous replication (replication_sync_timeout setting). I don't think it's been tested much.
Oh, you're using synchronous replication (replication_sync_timeout setting). I don't think it's been tested much.
FYI: This happens to me also when activating synchronous replication. Receiving the same oom error-message.
On Fri, Mar 24, 2017 at 2:52 PM, Timo Sirainen tss@iki.fi wrote:
On 24 Mar 2017, at 3.46, Daniel Miller dmiller@amfes.com wrote:
#14 replicator_queue_handle_sync_lookups (user=0xfa3fc0,
queue=0xf92260) at replicator-queue.c:278
Oh, you're using synchronous replication (replication_sync_timeout setting). I don't think it's been tested much.
On 3/24/2017 6:52 AM, Timo Sirainen wrote:
On 24 Mar 2017, at 3.46, Daniel Miller dmiller@amfes.com wrote:
#14 replicator_queue_handle_sync_lookups (user=0xfa3fc0, queue=0xf92260) at replicator-queue.c:278 Oh, you're using synchronous replication (replication_sync_timeout setting). I don't think it's been tested much.
Oh, that makes me feel so much better ;).
Ok...didn't realize I was breaking new ground here... I turned that setting on because I thought it would decrease the warnings about timeouts. Allright...let's see what turning it off does...
Daniel
participants (4)
-
Aki Tuomi
-
Christoph Kluge
-
Daniel Miller
-
Timo Sirainen