Dovecot 2.3.7.2 lucene segfault and coredump

Domenico Pastore domenico.pastore at par-tec.it
Thu Sep 19 13:11:33 EEST 2019


Hi,

i have find other information of the coredump.

My mail_plugin is:
mail_plugins = quota fts fts_lucene mail_log notify virtual

When I using quota and fts_lucene the coredump is generated.
If i use only the quota plugin or fts_lucene plugin, lmtp store work correctly.

Have you an ideas?

Br,
Domenico

Domenico Pastore
Senior Cloud Engineer
T 06.98269600 | M 347.1474270 | F 06.98269680
Par-Tec<http://www.par-tec.it> | beyond the IT domain
Via Cristoforo Colombo 163, 00147 Roma
CONFIDENZIALE: Questo messaggio ed i suoi allegati sono di carattere confidenziale. È vietato l'inoltro non autorizzato a destinatari diversi da quelli indicati nel messaggio originale. Se ricevuto per errore si prega di informare il mittente e cancellarlo immediatamente.
Il 16/09/19 17:20, Domenico Pastore via dovecot ha scritto:

Hi Stephan,

thanks for your replay.

This is bt full output:


(gdb) bt full
#0  clear (this=0x0) at /usr/src/debug/clucene-core-2.3.3.4/src/core/CLucene/util/VoidMap.h:123
No locals.
#1  lucene::search::FieldSortedHitQueue::_shutdown () at /usr/src/debug/clucene-core-2.3.3.4/src/core/CLucene/search/FieldSortedHitQueue.cpp:60
No locals.
#2  0x00007f7dc5eabe96 in _lucene_shutdown () at /usr/src/debug/clucene-core-2.3.3.4/src/core/CLucene/StdHeader.cpp:33
No locals.
#3  0x00007f7dc973bfbb in module_dir_deinit (modules=<optimized out>) at module-dir.c:578
        _data_stack_cur_id = 4
        module = 0x55ddb26268f0
        rev = 0x55ddb25bd050
        i = <optimized out>
        count = 6
#4  0x00007f7dc973c001 in module_dir_unload (modules=modules at entry=0x7f7dc9d2a380 <mail_storage_service_modules>) at module-dir.c:590
        module = <optimized out>
        next = <optimized out>
#5  0x00007f7dc9a26264 in mail_storage_service_deinit (_ctx=_ctx at entry=0x55ddb25d9848) at mail-storage-service.c:1730
        ctx = 0x55ddb2612c18
#6  0x000055ddb0e0dd35 in doveadm_mail_cmd_exec (ctx=ctx at entry=0x55ddb25d9808, wildcard_user=wildcard_user at entry=0x0) at doveadm-mail.c:673
        cctx = <optimized out>
        cli = <optimized out>
        ret = <optimized out>
        error = 0x7f7dc96762f7 <__x86_return_thunk+5> "\363\220\017\256\350\353\371H\215d$\b\303\350\a"
#7  0x000055ddb0e0ebbd in doveadm_cmd_ver2_to_mail_cmd_wrapper (cctx=<optimized out>) at doveadm-mail.c:1144
        mctx = 0x55ddb25d9808
        wildcard_user = <optimized out>
        fieldstr = 0x55ddb25b9318 "\315\"N\305}\177"
        pargv = {arr = {buffer = 0x55ddb25d9a38, element_size = 8}, v = 0x55ddb25d9a38, v_modifiable = 0x55ddb25d9a38}
        full_args = {arr = {buffer = <optimized out>, element_size = 8}, v = <optimized out>, v_modifiable = <optimized out>}
        i = <optimized out>
        cli = <optimized out>
        tcp_server = <optimized out>
        mail_cmd = {alloc = 0x7f7dc54e1d00 <cmd_quota_get_alloc>, name = 0x7f7dc54e22b6 "quota get", usage_args = 0x7f7dc54e2291 ""}
        dashdash = 0x55ddb0e4a7ca "--"
        args_pos = 1
#8  0x000055ddb0e1fc98 in doveadm_cmd_run_ver2 (argc=argc at entry=1, argv=argv at entry=0x55ddb25b9158, cctx=cctx at entry=0x7ffff7c52670) at doveadm-cmd.c:574
        param = <optimized out>
        pargv = {arr = {buffer = 0x55ddb25b9418, element_size = 104}, v = 0x55ddb25b9418, v_modifiable = 0x55ddb25b9418}
        opts = {arr = {buffer = <optimized out>, element_size = 32}, v = <optimized out>, v_modifiable = <optimized out>}
        pargc = 4
        c = -1
---Type <return> to continue, or q <return> to quit---
        li = -252221570
        pool = 0x55ddb25b9248
        optbuf = 0x55ddb25b9260
        __func__ = "doveadm_cmd_run_ver2"
#9  0x000055ddb0e24020 in doveadm_cmd_server_run_ver2 (cctx=0x7ffff7c52670, argv=0x55ddb25b9158, argc=1, conn=0x55ddb260dbf8) at client-connection-tcp.c:178
No locals.
#10 doveadm_cmd_handle (cctx=0x7ffff7c52670, argv=0x55ddb25b9158, argc=1, cmd_name=<optimized out>, conn=0x55ddb260dbf8) at client-connection-tcp.c:342
        mail_cmd = <optimized out>
        mctx = <optimized out>
        cmd_ver2 = <optimized out>
        prev_ioloop = 0x55ddb25c1cc0
        cmd = 0x0
#11 client_handle_command (args=<optimized out>, conn=0x55ddb260dbf8) at client-connection-tcp.c:415
        cctx = {cmd = 0x55ddb25d73e8, argc = 4, argv = 0x55ddb25b9450, username = 0x55ddb25b9119 "test4444client at newsvilpec.babel.it"<mailto:test4444client at newsvilpec.babel.it>, local_ip = {family = 2, u = {ip6 = {__in6_u = {
                  __u6_addr8 = "\300\240\n\003", '\000' <repeats 11 times>, __u6_addr16 = {41152, 778, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {51028160, 0, 0, 0}}}, ip4 = {s_addr = 51028160}}}, remote_ip = {family = 2, u = {
              ip6 = {__in6_u = {__u6_addr8 = "\n\000\b\200", '\000' <repeats 11 times>, __u6_addr16 = {10, 32776, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {2148007946, 0, 0, 0}}}, ip4 = {s_addr = 2148007946}}},
          local_port = 924, remote_port = 41491, conn_type = DOVEADM_CONNECTION_TYPE_TCP, input = 0x55ddb25d2dc0, output = 0x55ddb25d8d00}
        flags = <optimized out>
        cmd_name = <optimized out>
        argc = <optimized out>
#12 client_connection_tcp_input (conn=0x55ddb260dbf8) at client-connection-tcp.c:561
        args = <optimized out>
        _data_stack_cur_id = 3
        line = <optimized out>
        ok = true
        minor = 0
#13 0x00007f7dc973155f in io_loop_call_io (io=0x55ddb25e7b20) at ioloop.c:704
        ioloop = 0x55ddb25c1cc0
        t_id = 2
        __func__ = "io_loop_call_io"
#14 0x00007f7dc9732fa3 in io_loop_handler_run_internal (ioloop=ioloop at entry=0x55ddb25c1cc0) at ioloop-epoll.c:221
        ctx = 0x55ddb25c77b0
        events = <optimized out>
        list = 0x55ddb260be90
        io = <optimized out>
        tv = {tv_sec = 2147483, tv_usec = 0}
        events_count = <optimized out>
        msecs = <optimized out>
        ret = 1
        i = 0
---Type <return> to continue, or q <return> to quit---
        call = <optimized out>
        __func__ = "io_loop_handler_run_internal"
#15 0x00007f7dc9731666 in io_loop_handler_run (ioloop=ioloop at entry=0x55ddb25c1cc0) at ioloop.c:756
        __func__ = "io_loop_handler_run"
#16 0x00007f7dc9731888 in io_loop_run (ioloop=0x55ddb25c1cc0) at ioloop.c:729
        __func__ = "io_loop_run"
#17 0x00007f7dc96a42d3 in master_service_run (service=0x55ddb25c1b50, callback=callback at entry=0x55ddb0e27260 <client_connected>) at master-service.c:781
No locals.
#18 0x000055ddb0dfd421 in main (argc=1, argv=0x55ddb25c1890) at main.c:151
        set_roots = {0x7f7dc99c5d00 <master_service_ssl_setting_parser_info>, 0x55ddb106c6a0 <doveadm_setting_parser_info>, 0x0}
        input = {roots = 0x7ffff7c52850, config_path = 0x0, preserve_environment = false, preserve_user = false, preserve_home = false, never_exec = false, use_sysexits = false, parse_full_config = false,
          module = 0x55ddb0e46d3a "doveadm", service = 0x55ddb0e46d3a "doveadm", username = 0x0, local_ip = {family = 0, u = {ip6 = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0,
                    0}, __u6_addr32 = {0, 0, 0, 0}}}, ip4 = {s_addr = 0}}}, remote_ip = {family = 0, u = {ip6 = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {
                    0, 0, 0, 0}}}, ip4 = {s_addr = 0}}}, local_name = 0x0}
        output = {specific_services = 0x0, service_uses_local = false, service_uses_remote = false, used_local = false, used_remote = false, permission_denied = false}
        error = 0x7f7dc90ecca7 "__vdso_getcpu"
        c = <optimized out>
(gdb)


Br,

Domenico

Domenico Pastore
Senior Cloud Engineer
T 06.98269600 | M 347.1474270 | F 06.98269680
Par-Tec<http://www.par-tec.it> | beyond the IT domain
Via Cristoforo Colombo 163, 00147 Roma
CONFIDENZIALE: Questo messaggio ed i suoi allegati sono di carattere confidenziale. È vietato l'inoltro non autorizzato a destinatari diversi da quelli indicati nel messaggio originale. Se ricevuto per errore si prega di informare il mittente e cancellarlo immediatamente.
Il 16/09/19 17:05, Stephan Bosch ha scritto:


Op 16-9-2019 om 11:53 schreef Domenico Pastore via dovecot:

Hi,

I have other information.

I analyzed the core dump:

(gdb) core 22534
[New LWP 22534]
Reading symbols from /usr/libexec/dovecot/doveadm-server...Reading symbols from /usr/lib/debug/usr/libexec/dovecot/doveadm-server.debug...done.
done.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `dovecot-devel01/doveadm-server'.
Program terminated with signal 11, Segmentation fault.
#0  clear (this=0x0) at /usr/src/debug/clucene-core-2.3.3.4/src/core/CLucene/util/VoidMap.h:123
123            if ( dk || dv ){
(gdb)


Can you issue GDB command `bt full` at this point?

Regards,

Stephan.


Br,

Domenico

*Domenico Pastore*
Senior Cloud Engineer
*T* 06.98269600 | *M* 347.1474270 | *F* 06.98269680
*Par-Tec <http://www.par-tec.it><http://www.par-tec.it>* | *beyond the IT domain*
Via Cristoforo Colombo 163, 00147 Roma
*CONFIDENZIALE*: Questo messaggio ed i suoi allegati sono di carattere confidenziale. È vietato l'inoltro non autorizzato a destinatari diversi da quelli indicati nel messaggio originale. Se ricevuto per errore si prega di informare il mittente e cancellarlo immediatamente.

Il 16/09/19 09:50, Domenico Pastore via dovecot ha scritto:

Hi,

do you have the same problem with lucene?

I collected a strace on the dovecot process during indexing:

[pid 31339] execve("/usr/libexec/dovecot/doveadm-server", ["dovecot-devel01/doveadm-server"], [/* 18 vars */] <unfinished ...>
[pid 30711] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_KILLED, si_pid=30932, si_uid=0, si_status=SIGSEGV, si_utime=3, si_stime=1} ---
[pid 30711] write(62, "\0", 1)          = 1
[pid 30711] read(124, "8x\0\0008\0\0\0\251\3\0\0", 1020) = 12
[pid 30711] read(61, "\0", 64)          = 1
[pid 30711] open("/proc/sys/fs/suid_dumpable", O_RDONLY) = 227
[pid 30711] read(227, "0\n", 9)         = 2
[pid 30711] write(65, "25 30932 FATAL service(doveadm): child 30932 killed with signal 11 (core not dumped - https://dovecot.org/bugreport.html#coredumps - set /proc/sys/fs/suid_dumpable to 2)\n", 170) = 170
[pid 30711] write(14, "KILL\t30932\n", 11) = 11
[pid 30711] write(65, "25 30932 BYE\n", 13) = 13
[pid 31339] <... execve resumed> )      = 0
[pid 31339] open("/usr/lib64/dovecot/tls/x86_64/libz.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 31339] stat("/usr/lib64/dovecot/tls/x86_64", 0x7ffda7e3f1d0) = -1 ENOENT (No such file or directory)
[pid 31339] open("/usr/lib64/dovecot/tls/libz.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 31339] stat("/usr/lib64/dovecot/tls", 0x7ffda7e3f1d0) = -1 ENOENT (No such file or directory)
[pid 31339] open("/usr/lib64/dovecot/x86_64/libz.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 31339] stat("/usr/lib64/dovecot/x86_64", 0x7ffda7e3f1d0) = -1 ENOENT (No such file or directory)
[pid 31339] open("/usr/lib64/dovecot/libz.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 31339] stat("/usr/lib64/dovecot", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
[pid 31339] open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 4
[pid 31339] open("/lib64/libz.so.1", O_RDONLY|O_CLOEXEC) = 4


All missing libs are in "/usr/lib64" and no to "/usr/lib64/dovecot"

Can you help me?

Br,

Domenico



*Domenico Pastore*
Senior Cloud Engineer
*T* 06.98269600 | *M* 347.1474270 | *F* 06.98269680
*Par-Tec <http://www.par-tec.it><http://www.par-tec.it>* | *beyond the IT domain*
Via Cristoforo Colombo 163, 00147 Roma
*CONFIDENZIALE*: Questo messaggio ed i suoi allegati sono di carattere confidenziale. È vietato l'inoltro non autorizzato a destinatari diversi da quelli indicati nel messaggio originale. Se ricevuto per errore si prega di informare il mittente e cancellarlo immediatamente.

Il 12/09/19 12:07, Domenico Pastore via dovecot ha scritto:

Hi,

i have a problem with libclucene-core.so.2.3.3.4.
I See segfault error to messages log, when dovecot store mail to INBOX.

set 12 13:41:21 box02.par-tec.it kernel: doveadm-server[20285]: *segfault at 38 ip 00007f575b68fc42 sp 00007ffc0272e750 error 4 in libclucene-core.so.2.3.3.4*[7f575b569000+188000]
set 12 13:41:22 box02.par-tec.it kernel: doveadm-server[18871]: segfault at 38 ip 00007f9bea6d2c42 sp 00007fff47bceae0 error 4 in libclucene-core.so.2.3.3.4[7f9bea5ac000+188000]

In the dovecot logs:

2019-09-12 13:45:09 auth: Debug: ldap(test4444client at newsvilXXXX,10.0.8.128<mailto:test4444client at newsvilXXXX,10.0.8.128>): Finished userdb lookup
2019-09-12 13:45:09 auth: Debug: userdb out: USER    1 test4444client at newsvilXXXX home=/pec_store/pec_mailbox/pec_mailbox_devel01/4/5/6/7/1/45671_18231 quota_rule=*:bytes=1048576000    mail_plugins+=
2019-09-12 13:45:10  SKENO8Uvel0gbAAA5ZKy4w doveadm(test4444client at newsvilXXXX): *Fatal: master: service(doveadm): child 27680 killed with signal 11 (core dumped)*
2019-09-12 13:45:10 auth: Debug: master in: USER    1 test4444client at newsvilXXXX    service=doveadm lip=192.160.10.3    lport=924    rip=10.0.8.128 rport=36760
2019-09-12 13:45:10 auth: Debug: ldap(test4444client at newsvilXXXX 10.0.8.128): Performing userdb lookup

I have attach the coredump of error and dovecot.conf

Thanks for support.

Br,

Domenico


------------------- info: -------------------
# 2.3.7.2 (3c910f64b): /pec/config/dovecot/dovecot-devel01/dovecot.conf
# Pigeonhole version 0.5.7.2 (7372921a)
# OS: Linux 3.10.0-957.27.2.el7.x86_64 x86_64 CentOS Linux release 7.6.1810 (Core)
# Hostname: box02






-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://dovecot.org/pipermail/dovecot/attachments/20190919/75ea84c9/attachment-0001.html>


More information about the dovecot mailing list