Hi there,
I'm reporting what I think is a bug.
I just configured dovecot 2.4 with fts_solr plugin running on solr 9.8 release and it throws this panic and sigfaults one after another.
Mar 24 11:58:48 server dovecot: indexer-worker(hi@zakaria.website): Panic: settings_get() - event has no SETTINGS_EVENT_ROOT Mar 24 11:58:48 server dovecot: indexer-worker(hi@zakaria.website): Error: Raw backtrace: libdovecot.so.0(backtrace_append+0x39) [0x7f2fe692e549] -> libdovecot.so.0(backtrace_get+0x22) [0x7f2fe692e682] -> libdovecot.so.0(+0x13ccbc) [0x7f2fe693ccbc] -> libdovecot.so.0(+0x13cd51) [0x7f2fe693cd51] -> libdovecot.so.0(+0x6bfff) [0x7f2fe686bfff] -> libdovecot.so.0(+0x5ed08) [0x7f2fe685ed08] -> libdovecot.so.0(+0xba555) [0x7f2fe68ba555] -> libdovecot.so.0(settings_get+0x46) [0x7f2fe68bb576] -> libdovecot.so.0(ssl_client_settings_get+0x66) [0x7f2fe69196e6] -> libdovecot.so.0(http_client_init_ssl_ctx+0xa9) [0x7f2fe68d40a9] -> libdovecot.so.0(http_client_host_submit_request+0x9d) [0x7f2fe68d37dd] -> libdovecot.so.0(http_client_request_submit+0x34) [0x7f2fe68c70c4] -> libdovecot.so.0(+0xc7cf0) [0x7f2fe68c7cf0] -> libdovecot.so.0(http_client_request_send_payload+0x2e) [0x7f2fe68c7d3e] -> lib21_fts_solr_plugin.so(solr_connection_post_more+0x39) [0x7f2fe6e3af89] -> lib21_fts_solr_plugin.so(+0x4469) [0x7f2fe6e38469] -> lib20_fts_plugin.so(fts_backend_update_build_more+0x4a) [0x7f2fe6e5930a] -> lib20_fts_plugin.so(+0xa419) [0x7f2fe6e5a419] -> lib20_fts_plugin.so(+0xae8a) [0x7f2fe6e5ae8a] -> lib20_fts_plugin.so(fts_build_mail+0x57) [0x7f2fe6e5b827] -> lib20_fts_plugin.so(+0x119e1) [0x7f2fe6e619e1] -> libdovecot-storage.so.0(mail_precache+0x2c) [0x7f2fe6cd6b1c] -> indexer-worker(+0x3b8a) [0x559c6212fb8a] -> indexer-worker(+0x44b5) [0x559c621304b5] -> indexer-worker(+0x466b) [0x559c6213066b] -> libdovecot.so.0(connection_input_default+0x199) [0x7f2fe6933439] -> libdovecot.so.0(io_loop_call_io+0x67) [0x7f2fe69534d7] -> libdovecot.so.0(io_loop_handler_run_internal+0x12a) [0x7f2fe6954c0a] -> libdovecot.so.0(io_loop_handler_run+0x50) [0x7f2fe6953580] -> libdovecot.so.0(io_loop_run+0x48) [0x7f2fe6953758] Mar 24 11:58:48 server dovecot: indexer-worker(hi@zakaria.website): Fatal: master: service(indexer-worker): child 28216 killed with signal 6 (core dumps disabled - https://dovecot.org/bugreport.html#coredumps) Mar 24 11:59:13 server dovecot: decode2text: Error: cat: write error: Broken pipe Mar 24 11:59:13 server dovecot: decode2text: Error: /usr/libexec/dovecot/decode2text.sh: line 59: 29469 Done find . -name "$name" -print0 Mar 24 11:59:13 server dovecot: decode2text: Error: 29470 Done(123) | xargs -0 cat Mar 24 11:59:13 server dovecot: decode2text: Error: 29471 Segmentation fault | $libexec_dir/xml2text Mar 24 12:00:40 server dovecot: decode2text: Error: /usr/libexec/dovecot/decode2text.sh: line 59: 32540 Done find . -name "$name" -print0 Mar 24 12:00:40 server dovecot: decode2text: Error: 32541 | xargs -0 cat Mar 24 12:00:40 server dovecot: decode2text: Error: 32542 Segmentation fault | $libexec_dir/xml2text Mar 24 12:00:40 server dovecot: decode2text: Error: /usr/libexec/dovecot/decode2text.sh: line 59: 32555 Done find . -name "$name" -print0 Mar 24 12:00:40 server dovecot: decode2text: Error: 32556 | xargs -0 cat Mar 24 12:00:40 server dovecot: decode2text: Error: 32557 Segmentation fault | $libexec_dir/xml2text Mar 24 12:00:49 server dovecot: decode2text: Error: cat: write error: Broken pipe Mar 24 12:00:49 server dovecot: decode2text: Error: /usr/libexec/dovecot/decode2text.sh: line 59: 32832 Done find . -name "$name" -print0 Mar 24 12:00:49 server dovecot: decode2text: Error: 32833 Done(123) | xargs -0 cat Mar 24 12:00:49 server dovecot: decode2text: Error: 32834 Segmentation fault | $libexec_dir/xml2text Mar 24 12:00:52 server dovecot: imap-login: Login aborted: Connection closed (disconnected during TLS handshake) (tls_handshake_not_finished): user=, rip=20.65.195.124, lip=77.68.117.221, TLS handshaking: Connection closed, session= Mar 24 12:00:53 server dovecot: decode2text: Error: cat: write error: Broken pipe Mar 24 12:00:53 server dovecot: decode2text: Error: /usr/libexec/dovecot/decode2text.sh: line 59: 32955 Done find . -name "$name" -print0 Mar 24 12:00:53 server dovecot: decode2text: Error: 32956 Done(123) | xargs -0 cat Mar 24 12:00:53 server dovecot: decode2text: Error: 32957 Segmentation fault | $libexec_dir/xml2text Mar 24 12:00:57 server dovecot: decode2text: Error: cat: write error: Broken pipe Mar 24 12:00:57 server dovecot: decode2text: Error: /usr/libexec/dovecot/decode2text.sh: line 59: 33039 Done find . -name "$name" -print0 Mar 24 12:00:57 server dovecot: decode2text: Error: 33040 Done(123) | xargs -0 cat Mar 24 12:00:57 server dovecot: decode2text: Error: 33041 Segmentation fault | $libexec_dir/xml2text
These are my relevant fts configuration:-
mail_plugins = mail_compress fts fts_solr fts_autoindex = yes fts solr { url = SOLR_URL batch_size = 1000 } fts_decoder_driver = script fts_decoder_script_socket_path = decode2text service indexer-worker { vsz_limit = 4G } service decode2text { executable = script /usr/libexec/dovecot/decode2text.sh user = dovecot unix_listener decode2text { mode = 0666 } } language "en" { default = yes }
I think anyone can reproduce the panic using following commands while targeting a mailbox which contains as many emails as possible maybe 20K.
Commit and optimise dovecot core. /opt/solr-9.8.1/bin/solr api --solr-url "..../solr/dovecot/update?commit=true" /opt/solr-9.8.1/bin/solr api --solr-url "..../solr/dovecot/update?optimize=true"
Rescan and index the mailbox. doveadm fts rescan -u hi@zakaria.website doveadm index -u hi@zakaria.website -q "*"
I tested this on dovecot released solrconfig.xml for solr 9.
Note, current solr 9 has breaking changes to the config, I rectified using the following, please refer to https://solr.apache.org/guide/solr/latest/upgrade-notes/major-changes-in-sol....
Appending n solr deployment configuration following two lines:- SOLR_OPTS="$SOLR_OPTS -Dsolr.config.lib.enabled=true" SOLR_MODULES=extraction,ltr,analysis-extras
Also, additional note there are some other minor log warning, but weren't problematic.
With thanks.
Zak.
[https://api.zakaria.website/users/hi@zakaria.website/ f6ebf9ed031e93b5ed35f78ceb338278@zakaria.website/30db6adddb640dbc14bfad.png] Hi there,
I'm reporting what I think is a bug.
I just configured dovecot 2.4 with fts_solr plugin running on solr 9.8 release and it throws this panic and sigfaults one after another.
libdovecot.so.0(backtrace_get+0x22) [0x7f2fe692e682] -> libdovecot.so.0 (+0x13ccbc) [0x7f2fe693ccbc] -> libdovecot.so.0(+0x13cd51) [0x7f2fe693cd51] - libdovecot.so.0(+0x6bfff) [0x7f2fe686bfff] -> libdovecot.so.0(+0x5ed08) [0x7f2fe685ed08] -> libdovecot.so.0(+0xba555) [0x7f2fe68ba555] - libdovecot.so.0(settings_get+0x46) [0x7f2fe68bb576] -> libdovecot.so.0 (ssl_client_settings_get+0x66) [0x7f2fe69196e6] -> libdovecot.so.0 (http_client_init_ssl_ctx+0xa9) [0x7f2fe68d40a9] -> libdovecot.so.0 (http_client_host_submit_request+0x9d) [0x7f2fe68d37dd] -> libdovecot.so.0 (http_client_request_submit+0x34) [0x7f2fe68c70c4] -> libdovecot.so.0(+0xc7cf0) [0x7f2fe68c7cf0] -> libdovecot.so.0(http_client_request_send_payload+0x2e) [0x7f2fe68c7d3e] -> lib21_fts_solr_plugin.so(solr_connection_post_more+0x39) [0x7f2fe6e3af89] -> lib21_fts_solr_plugin.so(+0x4469) [0x7f2fe6e38469] - lib20_fts_plugin.so(fts_backend_update_build_more+0x4a) [0x7f2fe6e5930a] - lib20_fts_plugin.so(+0xa419) [0x7f2fe6e5a419] -> lib20_fts_plugin.so(+0xae8a) [0x7f2fe6e5ae8a] -> lib20_fts_plugin.so(fts_build_mail+0x57) [0x7f2fe6e5b827] - lib20_fts_plugin.so(+0x119e1) [0x7f2fe6e619e1] -> libdovecot-storage.so.0 (mail_precache+0x2c) [0x7f2fe6cd6b1c] -> indexer-worker(+0x3b8a) [0x559c6212fb8a] -> indexer-worker(+0x44b5) [0x559c621304b5] -> indexer-worker (+0x466b) [0x559c6213066b] -> libdovecot.so.0(connection_input_default+0x199) [0x7f2fe6933439] -> libdovecot.so.0(io_loop_call_io+0x67) [0x7f2fe69534d7] - libdovecot.so.0(io_loop_handler_run_internal+0x12a) [0x7f2fe6954c0a] - libdovecot.so.0(io_loop_handler_run+0x50) [0x7f2fe6953580] -> libdovecot.so.0 (io_loop_run+0x48) [0x7f2fe6953758] Mar 24 11:58:48 server dovecot: indexer-worker(hi@zakaria.website)<28216>: Fatal: master: service(indexer-worker): child 28216 killed with signal 6 (core dumps disabled - https://dovecot.org/bugreport.html#coredumps) Mar 24 11:59:13 server dovecot: decode2text: Error: cat: write error: Broken
Mar 24 11:58:48 server dovecot: indexer-worker(hi@zakaria.website)<28216>: Panic: settings_get() - event has no SETTINGS_EVENT_ROOT Mar 24 11:58:48 server dovecot: indexer-worker(hi@zakaria.website)<28216>: Error: Raw backtrace: libdovecot.so.0(backtrace_append+0x39) [0x7f2fe692e549] - pipe Mar 24 11:59:13 server dovecot: decode2text: Error: /usr/libexec/dovecot/ decode2text.sh: line 59: 29469 Done find . -name "$name" -print0 Mar 24 11:59:13 server dovecot: decode2text: Error: 29470 Done(123) | xargs - 0 cat Mar 24 11:59:13 server dovecot: decode2text: Error: 29471 Segmentation fault | $libexec_dir/xml2text Mar 24 12:00:40 server dovecot: decode2text: Error: /usr/libexec/dovecot/ decode2text.sh: line 59: 32540 Done find . -name "$name" -print0 Mar 24 12:00:40 server dovecot: decode2text: Error: 32541 | xargs -0 cat Mar 24 12:00:40 server dovecot: decode2text: Error: 32542 Segmentation fault | $libexec_dir/xml2text Mar 24 12:00:40 server dovecot: decode2text: Error: /usr/libexec/dovecot/ decode2text.sh: line 59: 32555 Done find . -name "$name" -print0 Mar 24 12:00:40 server dovecot: decode2text: Error: 32556 | xargs -0 cat Mar 24 12:00:40 server dovecot: decode2text: Error: 32557 Segmentation fault | $libexec_dir/xml2text Mar 24 12:00:49 server dovecot: decode2text: Error: cat: write error: Broken pipe Mar 24 12:00:49 server dovecot: decode2text: Error: /usr/libexec/dovecot/ decode2text.sh: line 59: 32832 Done find . -name "$name" -print0 Mar 24 12:00:49 server dovecot: decode2text: Error: 32833 Done(123) | xargs - 0 cat Mar 24 12:00:49 server dovecot: decode2text: Error: 32834 Segmentation fault | $libexec_dir/xml2text Mar 24 12:00:52 server dovecot: imap-login: Login aborted: Connection closed (disconnected during TLS handshake) (tls_handshake_not_finished): user=<>, rip=20.65.195.124, lip=77.68.117.221, TLS handshaking: Connection closed, session= Mar 24 12:00:53 server dovecot: decode2text: Error: cat: write error: Broken pipe Mar 24 12:00:53 server dovecot: decode2text: Error: /usr/libexec/dovecot/ decode2text.sh: line 59: 32955 Done find . -name "$name" -print0 Mar 24 12:00:53 server dovecot: decode2text: Error: 32956 Done(123) | xargs - 0 cat Mar 24 12:00:53 server dovecot: decode2text: Error: 32957 Segmentation fault | $libexec_dir/xml2text Mar 24 12:00:57 server dovecot: decode2text: Error: cat: write error: Broken pipe Mar 24 12:00:57 server dovecot: decode2text: Error: /usr/libexec/dovecot/ decode2text.sh: line 59: 33039 Done find . -name "$name" -print0 Mar 24 12:00:57 server dovecot: decode2text: Error: 33040 Done(123) | xargs - 0 cat Mar 24 12:00:57 server dovecot: decode2text: Error: 33041 Segmentation fault | $libexec_dir/xml2text
These are my relevant fts configuration:-
mail_plugins = mail_compress fts fts_solr fts_autoindex = yes fts solr { url = SOLR_URL batch_size = 1000 } fts_decoder_driver = script fts_decoder_script_socket_path = decode2text service indexer-worker { vsz_limit = 4G } service decode2text { executable = script /usr/libexec/dovecot/decode2text.sh user = dovecot unix_listener decode2text { mode = 0666 } } language "en" { default = yes }
I think anyone can reproduce the panic using following commands while targeting a mailbox which contains as many emails as possible maybe 20K.
Commit and optimise dovecot core. /opt/solr-9.8.1/bin/solr api --solr-url "..../solr/dovecot/update?commit=true" /opt/solr-9.8.1/bin/solr api --solr-url "..../solr/dovecot/ update?optimize=true"
Rescan and index the mailbox. doveadm fts rescan -u hi@zakaria.website doveadm index -u hi@zakaria.website -q "*"
I tested this on dovecot released solrconfig.xml for solr 9.
Note, current solr 9 has breaking changes to the config, I rectified using the following, please refer to https://solr.apache.org/guide/solr/latest/upgrade- notes/major-changes-in-solr-9.html.
Appending n solr deployment configuration following two lines:- SOLR_OPTS="$SOLR_OPTS -Dsolr.config.lib.enabled=true" SOLR_MODULES=extraction,ltr,analysis-extras
Also, additional note there are some other minor log warning, but weren't problematic.
With thanks.
Zak.