Indexer-worker panic and killed with signal 6 in 2.4 and Solr 9

25 Mar 2025


      Hi there,
I'm reporting what I think is a bug.
I just configured dovecot 2.4 with fts_solr plugin running on solr 9.8
release and it throws this panic and sigfaults one after another.
Mar 24 11:58:48 server dovecot: indexer-worker(hi@zakaria.website):
Panic: settings_get() - event has no SETTINGS_EVENT_ROOT
Mar 24 11:58:48 server dovecot: indexer-worker(hi@zakaria.website):
Error: Raw backtrace: libdovecot.so.0(backtrace_append+0x39)
[0x7f2fe692e549] -> libdovecot.so.0(backtrace_get+0x22) [0x7f2fe692e682]
-> libdovecot.so.0(+0x13ccbc) [0x7f2fe693ccbc] ->
libdovecot.so.0(+0x13cd51) [0x7f2fe693cd51] -> libdovecot.so.0(+0x6bfff)
[0x7f2fe686bfff] -> libdovecot.so.0(+0x5ed08) [0x7f2fe685ed08] ->
libdovecot.so.0(+0xba555) [0x7f2fe68ba555] ->
libdovecot.so.0(settings_get+0x46) [0x7f2fe68bb576] ->
libdovecot.so.0(ssl_client_settings_get+0x66) [0x7f2fe69196e6] ->
libdovecot.so.0(http_client_init_ssl_ctx+0xa9) [0x7f2fe68d40a9] ->
libdovecot.so.0(http_client_host_submit_request+0x9d) [0x7f2fe68d37dd]
-> libdovecot.so.0(http_client_request_submit+0x34) [0x7f2fe68c70c4] ->
libdovecot.so.0(+0xc7cf0) [0x7f2fe68c7cf0] ->
libdovecot.so.0(http_client_request_send_payload+0x2e) [0x7f2fe68c7d3e]
-> lib21_fts_solr_plugin.so(solr_connection_post_more+0x39)
[0x7f2fe6e3af89] -> lib21_fts_solr_plugin.so(+0x4469) [0x7f2fe6e38469]
-> lib20_fts_plugin.so(fts_backend_update_build_more+0x4a)
[0x7f2fe6e5930a] -> lib20_fts_plugin.so(+0xa419) [0x7f2fe6e5a419] ->
lib20_fts_plugin.so(+0xae8a) [0x7f2fe6e5ae8a] ->
lib20_fts_plugin.so(fts_build_mail+0x57) [0x7f2fe6e5b827] ->
lib20_fts_plugin.so(+0x119e1) [0x7f2fe6e619e1] ->
libdovecot-storage.so.0(mail_precache+0x2c) [0x7f2fe6cd6b1c] ->
indexer-worker(+0x3b8a) [0x559c6212fb8a] -> indexer-worker(+0x44b5)
[0x559c621304b5] -> indexer-worker(+0x466b) [0x559c6213066b] ->
libdovecot.so.0(connection_input_default+0x199) [0x7f2fe6933439] ->
libdovecot.so.0(io_loop_call_io+0x67) [0x7f2fe69534d7] ->
libdovecot.so.0(io_loop_handler_run_internal+0x12a) [0x7f2fe6954c0a] ->
libdovecot.so.0(io_loop_handler_run+0x50) [0x7f2fe6953580] ->
libdovecot.so.0(io_loop_run+0x48) [0x7f2fe6953758]
Mar 24 11:58:48 server dovecot: indexer-worker(hi@zakaria.website):
Fatal: master: service(indexer-worker): child 28216 killed with signal 6
(core dumps disabled - https://dovecot.org/bugreport.html#coredumps)
Mar 24 11:59:13 server dovecot: decode2text: Error: cat: write error:
Broken pipe
Mar 24 11:59:13 server dovecot: decode2text: Error:
/usr/libexec/dovecot/decode2text.sh: line 59: 29469 Done find . -name
"$name" -print0
Mar 24 11:59:13 server dovecot: decode2text: Error: 29470 Done(123) |
xargs -0 cat
Mar 24 11:59:13 server dovecot: decode2text: Error: 29471 Segmentation
fault | $libexec_dir/xml2text
Mar 24 12:00:40 server dovecot: decode2text: Error:
/usr/libexec/dovecot/decode2text.sh: line 59: 32540 Done find . -name
"$name" -print0
Mar 24 12:00:40 server dovecot: decode2text: Error: 32541 | xargs -0 cat
Mar 24 12:00:40 server dovecot: decode2text: Error: 32542 Segmentation
fault | $libexec_dir/xml2text
Mar 24 12:00:40 server dovecot: decode2text: Error:
/usr/libexec/dovecot/decode2text.sh: line 59: 32555 Done find . -name
"$name" -print0
Mar 24 12:00:40 server dovecot: decode2text: Error: 32556 | xargs -0 cat
Mar 24 12:00:40 server dovecot: decode2text: Error: 32557 Segmentation
fault | $libexec_dir/xml2text
Mar 24 12:00:49 server dovecot: decode2text: Error: cat: write error:
Broken pipe
Mar 24 12:00:49 server dovecot: decode2text: Error:
/usr/libexec/dovecot/decode2text.sh: line 59: 32832 Done find . -name
"$name" -print0
Mar 24 12:00:49 server dovecot: decode2text: Error: 32833 Done(123) |
xargs -0 cat
Mar 24 12:00:49 server dovecot: decode2text: Error: 32834 Segmentation
fault | $libexec_dir/xml2text
Mar 24 12:00:52 server dovecot: imap-login: Login aborted: Connection
closed (disconnected during TLS handshake) (tls_handshake_not_finished):
user=, rip=20.65.195.124, lip=77.68.117.221, TLS handshaking: Connection
closed, session=
Mar 24 12:00:53 server dovecot: decode2text: Error: cat: write error:
Broken pipe
Mar 24 12:00:53 server dovecot: decode2text: Error:
/usr/libexec/dovecot/decode2text.sh: line 59: 32955 Done find . -name
"$name" -print0
Mar 24 12:00:53 server dovecot: decode2text: Error: 32956 Done(123) |
xargs -0 cat
Mar 24 12:00:53 server dovecot: decode2text: Error: 32957 Segmentation
fault | $libexec_dir/xml2text
Mar 24 12:00:57 server dovecot: decode2text: Error: cat: write error:
Broken pipe
Mar 24 12:00:57 server dovecot: decode2text: Error:
/usr/libexec/dovecot/decode2text.sh: line 59: 33039 Done find . -name
"$name" -print0
Mar 24 12:00:57 server dovecot: decode2text: Error: 33040 Done(123) |
xargs -0 cat
Mar 24 12:00:57 server dovecot: decode2text: Error: 33041 Segmentation
fault | $libexec_dir/xml2text
These are my relevant fts configuration:-
mail_plugins = mail_compress fts fts_solr
fts_autoindex = yes
fts solr {
url = SOLR_URL
batch_size = 1000
}
fts_decoder_driver = script
fts_decoder_script_socket_path = decode2text
service indexer-worker {
vsz_limit = 4G
}
service decode2text {
executable = script /usr/libexec/dovecot/decode2text.sh
user = dovecot
unix_listener decode2text {
mode = 0666
}
}
language "en" {
default = yes
}
I think anyone can reproduce the panic using following commands while
targeting a mailbox which contains as many emails as possible maybe 20K.
Commit and optimise dovecot core.
/opt/solr-9.8.1/bin/solr api --solr-url
"..../solr/dovecot/update?commit=true"
/opt/solr-9.8.1/bin/solr api --solr-url
"..../solr/dovecot/update?optimize=true"
Rescan and index the mailbox.
doveadm fts rescan -u hi@zakaria.website
doveadm index -u hi@zakaria.website -q "*"
I tested this on dovecot released solrconfig.xml for solr 9.
Note, current solr 9 has breaking changes to the config, I rectified
using the following, please refer to
https://solr.apache.org/guide/solr/latest/upgrade-notes/major-changes-in-sol....
Appending n solr deployment configuration following two lines:-
SOLR_OPTS="$SOLR_OPTS -Dsolr.config.lib.enabled=true"
SOLR_MODULES=extraction,ltr,analysis-extras
Also, additional note there are some other minor log warning, but
weren't problematic.
With thanks.
Zak.
[https://api.zakaria.website/users/hi@zakaria.website/
f6ebf9ed031e93b5ed35f78ceb338278@zakaria.website/30db6adddb640dbc14bfad.png]
Hi there,
I'm reporting what I think is a bug.
I just configured dovecot 2.4 with fts_solr plugin running on solr 9.8 release
and it throws this panic and sigfaults one after another.
...
libdovecot.so.0(backtrace_get+0x22) [0x7f2fe692e682] -> libdovecot.so.0
(+0x13ccbc) [0x7f2fe693ccbc] -> libdovecot.so.0(+0x13cd51) [0x7f2fe693cd51] -
libdovecot.so.0(+0x6bfff) [0x7f2fe686bfff] -> libdovecot.so.0(+0x5ed08)
[0x7f2fe685ed08] -> libdovecot.so.0(+0xba555) [0x7f2fe68ba555] -
libdovecot.so.0(settings_get+0x46) [0x7f2fe68bb576] -> libdovecot.so.0
(ssl_client_settings_get+0x66) [0x7f2fe69196e6] -> libdovecot.so.0
(http_client_init_ssl_ctx+0xa9) [0x7f2fe68d40a9] -> libdovecot.so.0
(http_client_host_submit_request+0x9d) [0x7f2fe68d37dd] -> libdovecot.so.0
(http_client_request_submit+0x34) [0x7f2fe68c70c4] -> libdovecot.so.0(+0xc7cf0)
[0x7f2fe68c7cf0] -> libdovecot.so.0(http_client_request_send_payload+0x2e)
[0x7f2fe68c7d3e] -> lib21_fts_solr_plugin.so(solr_connection_post_more+0x39)
[0x7f2fe6e3af89] -> lib21_fts_solr_plugin.so(+0x4469) [0x7f2fe6e38469] -
lib20_fts_plugin.so(fts_backend_update_build_more+0x4a) [0x7f2fe6e5930a] -
lib20_fts_plugin.so(+0xa419) [0x7f2fe6e5a419] -> lib20_fts_plugin.so(+0xae8a)
[0x7f2fe6e5ae8a] -> lib20_fts_plugin.so(fts_build_mail+0x57) [0x7f2fe6e5b827] -
lib20_fts_plugin.so(+0x119e1) [0x7f2fe6e619e1] -> libdovecot-storage.so.0
(mail_precache+0x2c) [0x7f2fe6cd6b1c] -> indexer-worker(+0x3b8a)
[0x559c6212fb8a] -> indexer-worker(+0x44b5) [0x559c621304b5] -> indexer-worker
(+0x466b) [0x559c6213066b] -> libdovecot.so.0(connection_input_default+0x199)
[0x7f2fe6933439] -> libdovecot.so.0(io_loop_call_io+0x67) [0x7f2fe69534d7] -
libdovecot.so.0(io_loop_handler_run_internal+0x12a) [0x7f2fe6954c0a] -
libdovecot.so.0(io_loop_handler_run+0x50) [0x7f2fe6953580] -> libdovecot.so.0
(io_loop_run+0x48) [0x7f2fe6953758]
Mar 24 11:58:48 server dovecot: indexer-worker(hi@zakaria.website)<28216>:
Fatal: master: service(indexer-worker): child 28216 killed with signal 6 (core
dumps disabled - https://dovecot.org/bugreport.html#coredumps)
Mar 24 11:59:13 server dovecot: decode2text: Error: cat: write error: Broken
Mar 24 11:58:48 server dovecot: indexer-worker(hi@zakaria.website)<28216>:
Panic: settings_get() - event has no SETTINGS_EVENT_ROOT
Mar 24 11:58:48 server dovecot: indexer-worker(hi@zakaria.website)<28216>:
Error: Raw backtrace: libdovecot.so.0(backtrace_append+0x39) [0x7f2fe692e549] -
pipe
Mar 24 11:59:13 server dovecot: decode2text: Error: /usr/libexec/dovecot/
decode2text.sh: line 59: 29469 Done find . -name "$name" -print0
Mar 24 11:59:13 server dovecot: decode2text: Error: 29470 Done(123) | xargs -
0 cat
Mar 24 11:59:13 server dovecot: decode2text: Error: 29471 Segmentation fault |
$libexec_dir/xml2text
Mar 24 12:00:40 server dovecot: decode2text: Error: /usr/libexec/dovecot/
decode2text.sh: line 59: 32540 Done find . -name "$name" -print0
Mar 24 12:00:40 server dovecot: decode2text: Error: 32541 | xargs -0 cat
Mar 24 12:00:40 server dovecot: decode2text: Error: 32542 Segmentation fault |
$libexec_dir/xml2text
Mar 24 12:00:40 server dovecot: decode2text: Error: /usr/libexec/dovecot/
decode2text.sh: line 59: 32555 Done find . -name "$name" -print0
Mar 24 12:00:40 server dovecot: decode2text: Error: 32556 | xargs -0 cat
Mar 24 12:00:40 server dovecot: decode2text: Error: 32557 Segmentation fault |
$libexec_dir/xml2text
Mar 24 12:00:49 server dovecot: decode2text: Error: cat: write error: Broken
pipe
Mar 24 12:00:49 server dovecot: decode2text: Error: /usr/libexec/dovecot/
decode2text.sh: line 59: 32832 Done find . -name "$name" -print0
Mar 24 12:00:49 server dovecot: decode2text: Error: 32833 Done(123) | xargs -
0 cat
Mar 24 12:00:49 server dovecot: decode2text: Error: 32834 Segmentation fault |
$libexec_dir/xml2text
Mar 24 12:00:52 server dovecot: imap-login: Login aborted: Connection closed
(disconnected during TLS handshake) (tls_handshake_not_finished): user=<>,
rip=20.65.195.124, lip=77.68.117.221, TLS handshaking: Connection closed,
session=
Mar 24 12:00:53 server dovecot: decode2text: Error: cat: write error: Broken
pipe
Mar 24 12:00:53 server dovecot: decode2text: Error: /usr/libexec/dovecot/
decode2text.sh: line 59: 32955 Done find . -name "$name" -print0
Mar 24 12:00:53 server dovecot: decode2text: Error: 32956 Done(123) | xargs -
0 cat
Mar 24 12:00:53 server dovecot: decode2text: Error: 32957 Segmentation fault |
$libexec_dir/xml2text
Mar 24 12:00:57 server dovecot: decode2text: Error: cat: write error: Broken
pipe
Mar 24 12:00:57 server dovecot: decode2text: Error: /usr/libexec/dovecot/
decode2text.sh: line 59: 33039 Done find . -name "$name" -print0
Mar 24 12:00:57 server dovecot: decode2text: Error: 33040 Done(123) | xargs -
0 cat
Mar 24 12:00:57 server dovecot: decode2text: Error: 33041 Segmentation fault |
$libexec_dir/xml2text
These are my relevant fts configuration:-
mail_plugins = mail_compress fts fts_solr
fts_autoindex = yes
fts solr {
url = SOLR_URL
batch_size = 1000
}
fts_decoder_driver = script
fts_decoder_script_socket_path = decode2text
service indexer-worker {
vsz_limit = 4G
}
service decode2text {
executable = script /usr/libexec/dovecot/decode2text.sh
user = dovecot
unix_listener decode2text {
mode = 0666
}
}
language "en" {
default = yes
}
I think anyone can reproduce the panic using following commands while targeting
a mailbox which contains as many emails as possible maybe 20K.
Commit and optimise dovecot core.
/opt/solr-9.8.1/bin/solr api --solr-url "..../solr/dovecot/update?commit=true"
/opt/solr-9.8.1/bin/solr api --solr-url "..../solr/dovecot/
update?optimize=true"
Rescan and index the mailbox.
doveadm fts rescan -u hi@zakaria.website
doveadm index -u hi@zakaria.website -q "*"
I tested this on dovecot released solrconfig.xml for solr 9.
Note, current solr 9 has breaking changes to the config, I rectified using the
following, please refer to https://solr.apache.org/guide/solr/latest/upgrade-
notes/major-changes-in-solr-9.html.
Appending n solr deployment configuration following two lines:-
SOLR_OPTS="$SOLR_OPTS -Dsolr.config.lib.enabled=true"
SOLR_MODULES=extraction,ltr,analysis-extras
Also, additional note there are some other minor log warning, but weren't
problematic.
With thanks.
Zak.

Zakaria

Aki Tuomi

Zakaria

tags

participants (2)