Aw: bug in indexer/indexer-worker
Infoomatic
infoomatic at gmx.at
Tue Jun 16 13:42:03 UTC 2015
I guess this is the same issue: http://www.dovecot.org/list/dovecot/2015-June/101029.html
> Gesendet: Dienstag, 16. Juni 2015 um 14:07 Uhr
> Von: Infoomatic <infoomatic at gmx.at>
> An: "Dovecot ML" <dovecot at dovecot.org>
> Betreff: bug in indexer/indexer-worker
>
> Hi,
>
> I have already mentioned this in http://www.dovecot.org/pipermail/dovecot/2014-November/098592.html
>
> I could reproduce the errors above in a self-compiled v2.2.18 and the prebuilt packages from xi.rename-it.nl (in addition to version 2.2.15 mentioned in the link)
>
> The problem occurs when enabling fts_tika in the plugins (tried tika 1.6, 1.7 and 1.8). I tried to move a folder of my mailbox with about 2000 mails to my server (no users, modern hardware).
> At some point I get an error and from this time on dovecot keeps repeating the last lines with every mail that comes in ... see [1]
> I then get kernel syslog messages like in [2]. when I try to index all users (had another testuser) mailboxes with "doveadm index" I get the errors of [3].
>
> The mailfolder does contain mails with attachments of all sorts, videos, office documents etc. When disabling fts_tika it works. dovecot -n output is [4]
>
> Could someone of the devs please have a look?
>
> best regards,
> infoomatic
>
>
> [1] Then comes the panic which produces loads of errormessages below the "Raw backtrace" part:
> Jun 16 10:11:08 testhost dovecot: indexer-worker(user1): Warning: I/O leak: 0x7f40b55c39d0 (line 411, fd 20)
> Jun 16 10:11:08 testhost dovecot: indexer-worker(user1): Warning: Timeout leak: 0x7f40b55c3780 (line 325)
> Jun 16 10:11:08 testhost dovecot: indexer-worker(user1): Panic: file ioloop.c: line 39 (io_add_file): assertion failed: (callback != NULL)
> Jun 16 10:11:08 testhost dovecot: indexer-worker(user1): Error: Raw backtrace: /opt/dovecot/lib/dovecot/libdovecot.so.0(+0x78320) [0x7f40b55f4320] -> /opt/dovecot/lib/dovecot/libdovecot.so.0(+0x783fe) [0x7f40b55f43fe] -> /opt/dovecot/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7f40b55a3b25] -> /opt/dovecot/lib/dovecot/libdovecot.so.0(+0x897d0) [0x7f40b56057d0] -> /opt/dovecot/lib/dovecot/libdovecot.so.0(io_loop_move_io+0x3e) [0x7f40b560643e] -> /opt/dovecot/lib/dovecot/libdovecot.so.0(connection_switch_ioloop+0x14) [0x7f40b55f21d4] -> /opt/dovecot/lib/dovecot/libdovecot.so.0(http_client_switch_ioloop+0x28) [0x7f40b55c8148] -> /opt/dovecot/lib/dovecot/libdovecot.so.0(+0x44536) [0x7f40b55c0536] -> /opt/dovecot/lib/dovecot/lib21_fts_solr_plugin.so(solr_connection_post_more+0x49) [0x7f40b108d469] -> /opt/dovecot/lib/dovecot/lib21_fts_solr_plugin.so(+0x3ac9) [0x7f40b1089ac9] -> /opt/dovecot/lib/dovecot/lib21_fts_solr_plugin.so(+0x3b6d) [0x7f40b1089b6d] -> /opt/dovecot/lib/dovecot/lib20_fts_plugin.so(+0x86ec) [0x7f40b495c6ec] -> /opt/dovecot/lib/dovecot/lib20_fts_plugin.so(fts_backend_update_deinit+0x23) [0x7f40b495c723] -> /opt/dovecot/lib/dovecot/lib20_fts_plugin.so(+0xe5cc) [0x7f40b49625cc] -> /opt/dovecot/lib/dovecot/lib20_fts_plugin.so(+0xf451) [0x7f40b4963451] -> /opt/dovecot/lib/dovecot/lib10_quota_plugin.so(+0xb454) [0x7f40b4b97454] -> /opt/dovecot/lib/dovecot/lib01_acl_plugin.so(+0xcfea) [0x7f40b4daafea] -> /opt/dovecot/lib/dovecot/libdovecot-storage.so.0(mailbox_transaction_commit_get_changes+0x41) [0x7f40b58a64f1] -> /opt/dovecot/lib/dovecot/libdovecot-storage.so.0(mailbox_transaction_commit+0xe) [0x7f40b58a658e] -> dovecot/indexer-worker() [0x40250c] -> /opt/dovecot/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x4c) [0x7f40b560615c] -> /opt/dovecot/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0xbb) [0x7f40b56070ab] -> /opt/dovecot/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x9) [0x7f40b56061c9] -> /opt/dovecot/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7f40b5606248] -> /opt/dovecot/lib/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7f40b55a9093] -> dovecot/indexer-worker(main+0xe3) [0x401f03] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f40b51d8ec5] -> dovecot/indexer-worker() [0x401fc2]
> Jun 16 10:11:08 testhost dovecot: indexer: Error: Indexer worker disconnected, discarding 2 requests for user1
> Jun 16 10:11:08 testhost dovecot: indexer-worker(user1): Fatal: master: service(indexer-worker): child 2066 killed with signal 6 (core dumped)
> Jun 16 10:11:09 testhost dovecot: indexer-worker(user1): Warning: I/O leak: 0x7f25bc1689d0 (line 120, fd 23)
> Jun 16 10:11:09 testhost dovecot: indexer-worker(user1): Warning: Timeout leak: 0x7f25bc168780 (line 325)
> Jun 16 10:11:09 testhost dovecot: indexer: Error: Indexer worker disconnected, discarding 2 requests for user1
> Jun 16 10:11:09 testhost dovecot: indexer-worker(user1): Fatal: master: service(indexer-worker): child 2124 killed with signal 11 (core dumped)
> Jun 16 10:11:09 testhost dovecot: indexer-worker(user1): Warning: I/O leak: 0x7f6cabb149d0 (line 120, fd 23)
> Jun 16 10:11:09 testhost dovecot: indexer-worker(user1): Warning: Timeout leak: 0x7f6cabb14780 (line 325)
> Jun 16 10:11:09 testhost dovecot: indexer: Error: Indexer worker disconnected, discarding 2 requests for user1
> Jun 16 10:11:09 testhost dovecot: indexer-worker(user1): Fatal: master: service(indexer-worker): child 2125 killed with signal 11 (core dumped)
>
> [2] kernel syslog:
> [Tue Jun 16 09:51:15 2015] indexer-worker[7592]: segfault at 8 ip 00007f066e8c8c27 sp 00007fff36a038b0 error 4 in libdovecot.so.0.0.0[7f066e832000+e7000]
> [Tue Jun 16 09:51:16 2015] indexer-worker[7594]: segfault at 8 ip 00007f702671ac27 sp 00007ffc6e134840 error 4 in libdovecot.so.0.0.0[7f7026684000+e7000]
> [Tue Jun 16 09:51:16 2015] indexer-worker[7596]: segfault at 20 ip 00007fb8a5af2c47 sp 00007fff0a344780 error 4 in libdovecot.so.0.0.0[7fb8a5a69000+e7000]
> [Tue Jun 16 09:51:17 2015] indexer-worker[7598]: segfault at 20 ip 00007f5834b6ac47 sp 00007ffd54f35c00 error 4 in libdovecot.so.0.0.0[7f5834ae1000+e7000]
> [Tue Jun 16 09:51:18 2015] indexer-worker[7600]: segfault at 20 ip 00007f5b2f6fac47 sp 00007fff1a697690 error 4 in libdovecot.so.0.0.0[7f5b2f671000+e7000]
> [Tue Jun 16 09:51:19 2015] indexer-worker[7602]: segfault at 20 ip 00007fd6084ddc47 sp 00007ffdb4495af0 error 4 in libdovecot.so.0.0.0[7fd608454000+e7000]
> [Tue Jun 16 09:51:20 2015] indexer-worker[7604]: segfault at 20 ip 00007f7049789c47 sp 00007ffe71e4bdd0 error 4 in libdovecot.so.0.0.0[7f7049700000+e7000]
> [Tue Jun 16 09:51:20 2015] indexer-worker[7606]: segfault at 20 ip 00007f771d54dc47 sp 00007ffe01c6ff80 error 4 in libdovecot.so.0.0.0[7f771d4c4000+e7000]
> [Tue Jun 16 09:51:21 2015] indexer-worker[7608]: segfault at 20 ip 00007fc19d482c47 sp 00007ffc9a0b0280 error 4 in libdovecot.so.0.0.0[7fc19d3f9000+e7000]
> [Tue Jun 16 09:51:22 2015] indexer-worker[7610]: segfault at 29 ip 00007fa14df25c2b sp 00007ffeb246e5f0 error 4 in libdovecot.so.0.0.0[7fa14de8f000+e7000]
> [Tue Jun 16 09:51:23 2015] indexer-worker[7612]: segfault at 8 ip 00007fe8e41f2c27 sp 00007ffc700e2170 error 4 in libdovecot.so.0.0.0[7fe8e415c000+e7000]
> [Tue Jun 16 09:51:24 2015] indexer-worker[7614]: segfault at 8 ip 00007ff9a0c76c27 sp 00007fffecd44c90 error 4 in libdovecot.so.0.0.0[7ff9a0be0000+e7000]
> [Tue Jun 16 09:51:25 2015] indexer-worker[7616]: segfault at 8 ip 00007fd27867dc27 sp 00007ffedd8407f0 error 4 in libdovecot.so.0.0.0[7fd2785e7000+e7000]
> [Tue Jun 16 09:51:25 2015] indexer-worker[7618]: segfault at 8 ip 00007f94f8753c27 sp 00007fff56cc5ea0 error 4 in libdovecot.so.0.0.0[7f94f86bd000+e7000]
>
> [3] errors from "doveadm index":
> /opt/dovecot/bin/doveadm index -A '*'
> doveadm(user1): Warning: I/O leak: 0x7f0e072519d0 (line 120, fd 27)
> doveadm(user1): Warning: Timeout leak: 0x7f0e07251780 (line 325)
> Segmentation fault (core dumped)
>
> /opt/dovecot/bin/doveadm index -u user1 '*'
> doveadm(user1): Warning: I/O leak: 0x7f99b4fba9d0 (line 411, fd 14)
> doveadm(user1): Warning: Timeout leak: 0x7f99b4fba780 (line 325)
> doveadm(user1): Fatal: pool_system_realloc(17179869184): Out of memory
> doveadm(user1): Error: Raw backtrace: /opt/dovecot/lib/dovecot/libdovecot.so.0(+0x78320) [0x7f99b4feb320] -> /opt/dovecot/lib/dovecot/libdovecot.so.0(default_fatal_handler+0x2a) [0x7f99b4feb38a] -> /opt/dovecot/lib/dovecot/libdovecot.so.0(i_error+0) [0x7f99b4f9ac7f] -> /opt/dovecot/lib/dovecot/libdovecot.so.0(+0x8f3a3) [0x7f99b50023a3] -> /opt/dovecot/lib/dovecot/libdovecot.so.0(+0x7472d) [0x7f99b4fe772d] -> /opt/dovecot/lib/dovecot/libdovecot.so.0(buffer_get_space_unsafe+0x5f) [0x7f99b4fe7f5f] -> /opt/dovecot/lib/dovecot/libdovecot.so.0(io_loop_handle_add+0x24) [0x7f99b4ffdd94] -> /opt/dovecot/lib/dovecot/libdovecot.so.0(+0x89761) [0x7f99b4ffc761] -> /opt/dovecot/lib/dovecot/libdovecot.so.0(io_loop_move_io+0x3e) [0x7f99b4ffd43e] -> /opt/dovecot/lib/dovecot/libdovecot.so.0(connection_switch_ioloop+0x14) [0x7f99b4fe91d4] -> /opt/dovecot/lib/dovecot/libdovecot.so.0(http_client_switch_ioloop+0x28) [0x7f99b4fbf148] -> /opt/dovecot/lib/dovecot/libdovecot.so.0(+0x44536) [0x7f99b4fb7536] -> /opt/dovecot/lib/dovecot/lib21_fts_solr_plugin.so(solr_connection_post_more+0x49) [0x7f99b0c9d469] -> /opt/dovecot/lib/dovecot/lib21_fts_solr_plugin.so(+0x373f) [0x7f99b0c9973f] -> /opt/dovecot/lib/dovecot/lib20_fts_plugin.so(+0x91e8) [0x7f99b43541e8] -> /opt/dovecot/lib/dovecot/lib20_fts_plugin.so(+0x95f0) [0x7f99b43545f0] -> /opt/dovecot/lib/dovecot/lib20_fts_plugin.so(fts_build_mail+0x714) [0x7f99b4354d34] -> /opt/dovecot/lib/dovecot/lib20_fts_plugin.so(+0xf17e) [0x7f99b435a17e] -> /opt/dovecot/lib/dovecot/libdovecot-storage.so.0(mail_precache+0x19) [0x7f99b5294e89] -> /opt/dovecot/bin/doveadm() [0x423211] -> /opt/dovecot/bin/doveadm() [0x41ecc6] -> /opt/dovecot/bin/doveadm(doveadm_mail_try_run+0x27f) [0x41fb8f] -> /opt/dovecot/bin/doveadm(main+0x38f) [0x412a9f] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f99b4bcfec5] -> /opt/dovecot/bin/doveadm() [0x412b99]
>
> [4] dovecot -n
> # 2.2.18: /opt/dovecot/etc/dovecot/dovecot.conf
> # Pigeonhole version 0.4.8 (0c4ae064f307+)
> # OS: Linux 3.13.0-52-generic x86_64 Ubuntu 14.04.2 LTS ext4
> auth_username_format = %n
> auth_verbose = yes
> disable_plaintext_auth = no
> imap_idle_notify_interval = 5 mins
> mail_gid = vmail
> mail_home = /var/lib/vmail/home/%d/%u
> mail_location = mdbox:/var/lib/vmail/mail/%d/%u
> mail_plugins = fts fts_solr zlib acl quota zlib fts fts_solr acl
> mail_uid = vmail
> mailbox_idle_check_interval = 1 mins
> managesieve_notify_capability = mailto
> managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate imapflags notify
> mdbox_rotate_size = 32 M
> namespace {
> list = children
> location = mdbox:/var/lib/vmail/mail/%d/%%u
> prefix = shared/%%u/
> separator = /
> subscriptions = no
> type = shared
> }
> namespace inbox {
> inbox = yes
> location =
> mailbox Drafts {
> special_use = \Drafts
> }
> mailbox Junk {
> special_use = \Junk
> }
> mailbox Sent {
> special_use = \Sent
> }
> mailbox "Sent Messages" {
> special_use = \Sent
> }
> mailbox Trash {
> special_use = \Trash
> }
> prefix =
> separator = /
> type = private
> }
> passdb {
> args = /opt/dovecot/etc/dovecot/dovecot-ldap.conf.ext
> driver = ldap
> }
> plugin {
> acl = vfile
> acl_defaults_from_inbox = yes
> acl_shared_dict = file:/var/lib/vmail/shared-mailboxes
> fts = solr
> fts_autoindex = yes
> fts_solr = url=http://localhost:8983/solr/ break-imap-search
> fts_tika = http://localhost:8081/tika
> quota = dict:Quota::file:%h/dovecot-quota
> quota_rule = *:storage=1G
> quota_warning = storage=95%% quota-warning 95 %u
> quota_warning2 = storage=80%% quota-warning 80 %u
> sieve = ~/.dovecot.sieve
> sieve_dir = ~/sieve
> sieve_extensions = +notify +imapflags
> zlib_save = gz
> zlib_save_level = 9
> }
> protocols = imap sieve
> service auth {
> unix_listener /var/spool/postfix/private/auth {
> group = postfix
> mode = 0666
> user = postfix
> }
> unix_listener auth-userdb {
> group = vmail
> user = vmail
> }
> }
> service imap-login {
> inet_listener imaps {
> port = 0
> }
> }
> service managesieve-login {
> inet_listener sieve_deprecated {
> port = 2000
> }
> }
> service pop3-login {
> inet_listener pop3 {
> port = 0
> }
> inet_listener pop3s {
> port = 0
> }
> }
> ssl = required
> ssl_cert = </etc/ssl/testhost/testhost.crt
> ssl_cipher_list = EDH+CAMELLIA:EDH+aRSA:EECDH+aRSA+AESGCM:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH:+CAMELLIA256:+AES256:+CAMELLIA128:+AES128:+SSLv3:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!DSS:!RC4:!SEED:!ECDSA:CAMELLIA256-SHA:AES256-SHA:CAMELLIA128-SHA:AES128-SHA
> ssl_key = </etc/ssl/testhost/testhost.key
> ssl_prefer_server_ciphers = yes
> ssl_protocols = !SSLv2 !SSLv3
> userdb {
> args = /opt/dovecot/etc/dovecot/dovecot-ldap.conf.ext
> driver = ldap
> }
> protocol lda {
> mail_plugins = fts fts_solr zlib sieve acl
> }
> protocol imap {
> mail_plugins = fts fts_solr zlib acl quota imap_acl imap_quota
> }
> protocol sieve {
> mail_plugins = fts fts_solr
> }
>
More information about the dovecot
mailing list