<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<div class="moz-cite-prefix">Il 31/05/21 21:00, Aki Tuomi ha
scritto:<br>
</div>
<blockquote type="cite"
cite="mid:533954324.3804.1622487611836@appsuite-dev-gw2.open-xchange.com">
<pre class="moz-quote-pre" wrap="">
</pre>
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">On 31/05/2021 21:58 Alessio Cecchi <a class="moz-txt-link-rfc2396E" href="mailto:alessio@skye.it"><alessio@skye.it></a> wrote:
Il 31/05/21 18:17, Aki Tuomi ha scritto:
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">On 31/05/2021 19:09 Alessio Cecchi <a class="moz-txt-link-rfc2396E" href="mailto:alessio@skye.it"><alessio@skye.it></a> wrote:
Hi,
I have setup on a (little busy) Dovecot server FTS with Solr and Virtual folder to enable "search in all folders" for users.
All works fine until for some users the indexer-worker process crash.
After this crash Dovecot stop to query Solr for new search in BODY, returning SERVERBUG, for all users on the server and only with a dovecot restart users can search in BODY again.
Before the dovecot restart I can see that on Solr server no query from dovecot arriving anymore (for SEARCH BODY because SEARCH in FROM/TO/SUBJECT still works fine)
</pre>
</blockquote>
<pre class="moz-quote-pre" wrap="">Hi!
It's a known bug and will be fixed in next release. It happens with tika and solr being used together.
Aki
</pre>
</blockquote>
<pre class="moz-quote-pre" wrap="">Good to know, I'm not using tika but decode2text, is the same?
</pre>
</blockquote>
<pre class="moz-quote-pre" wrap="">
Hm. Not necessarely. It could be something related to some other http connection though. The fix done should help in any case.
Aki
</pre>
</blockquote>
<p>Hi,</p>
<p>I'm not sure if can help but I have collect the core dump also
for this crash. I notice that happens only when indexing Virtual
mailbox folder, example:</p>
<p>- doveadm index -u <a class="moz-txt-link-abbreviated" href="mailto:user4@email.com">user4@email.com</a> '*' works fine</p>
<p>- doveadm index -u <a class="moz-txt-link-abbreviated" href="mailto:user4@email.com">user4@email.com</a> 'Virtual/All' crash</p>
<p>If I run first the index on '*' and after on 'Virtual/All' works
fine for both. But when users from webmail search in all folders
via Virtual/All always crash and after no more indexer-worker run
until I reload dovecot or kill "indexer" process.<br>
</p>
<p>Here the core dump:</p>
<p>Jun 03 05:05:34 Panic: indexer-worker(<a class="moz-txt-link-abbreviated" href="mailto:user4@email.com">user4@email.com</a>)
session=<x0EZ2dPDiuW5Ydlx:pQBBH/tGuGDjBwAAxbYlJw>: file
http-client-request.c: line 1240 (http_client_request_send_more):
assertion failed: (req->payload_input != NULL)<br>
Jun 03 05:05:34 Error: indexer-worker(<a class="moz-txt-link-abbreviated" href="mailto:user4@email.com">user4@email.com</a>)
session=<x0EZ2dPDiuW5Ydlx:pQBBH/tGuGDjBwAAxbYlJw>: Raw
backtrace:
/usr/lib64/dovecot/libdovecot.so.0(backtrace_append+0x42)
[0x7f5d132cbac2] ->
/usr/lib64/dovecot/libdovecot.so.0(backtrace_get+0x1e)
[0x7f5d132cbbce] ->
/usr/lib64/dovecot/libdovecot.so.0(+0xf3cde) [0x7f5d132d8cde]
-> /usr/lib64/dovecot/libdovecot.so.0(+0xf3d81)
[0x7f5d132d8d81] ->
/usr/lib64/dovecot/libdovecot.so.0(i_fatal+0) [0x7f5d1322a25a]
->
/usr/lib64/dovecot/libdovecot.so.0(http_client_request_send_more+0x3dd)
[0x7f5d132729ad] ->
/usr/lib64/dovecot/libdovecot.so.0(http_client_connection_output+0xf1)
[0x7f5d13277101] ->
/usr/lib64/dovecot/libdovecot.so.0(+0x11d2b0) [0x7f5d133022b0]
-> /usr/lib64/dovecot/libdovecot.so.0(io_loop_call_io+0x65)
[0x7f5d132f0f45] ->
/usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x12b)
[0x7f5d132f28fb] ->
/usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run+0x59)
[0x7f5d132f1049] ->
/usr/lib64/dovecot/libdovecot.so.0(io_loop_run+0x38)
[0x7f5d132f1288] ->
/usr/lib64/dovecot/libdovecot.so.0(http_client_wait+0xcd)
[0x7f5d1328020d] ->
/usr/lib64/dovecot/lib21_fts_solr_plugin.so(solr_connection_select+0xe4)
[0x7f5d11440174] ->
/usr/lib64/dovecot/lib21_fts_solr_plugin.so(+0x45d4)
[0x7f5d1143c5d4] ->
/usr/lib64/dovecot/lib20_fts_plugin.so(fts_backend_get_last_uid+0x6e)
[0x7f5d125af3fe] ->
/usr/lib64/dovecot/lib20_fts_plugin.so(+0xf952) [0x7f5d125b5952]
-> /usr/lib64/dovecot/lib20_fts_plugin.so(+0x10ef6)
[0x7f5d125b6ef6] ->
/usr/lib64/dovecot/lib20_virtual_plugin.so(+0x966a)
[0x7f5d1239b66a] ->
/usr/lib64/dovecot/lib20_fts_plugin.so(+0x10ba6) [0x7f5d125b6ba6]
->
/usr/lib64/dovecot/libdovecot-storage.so.0(mail_precache+0x2e)
[0x7f5d135dbb0e] -> dovecot/indexer-worker [<a class="moz-txt-link-abbreviated" href="mailto:user4@email.com">user4@email.com</a>
Virtual/All](+0x2924) [0x561e504cb924] ->
/usr/lib64/dovecot/libdovecot.so.0(io_loop_call_io+0x65)
[0x7f5d132f0f45] ->
/usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x12b)
[0x7f5d132f28fb] ->
/usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run+0x59)
[0x7f5d132f1049] ->
/usr/lib64/dovecot/libdovecot.so.0(io_loop_run+0x38)
[0x7f5d132f1288] ->
/usr/lib64/dovecot/libdovecot.so.0(master_service_run+0x13)
[0x7f5d1325bbb3] -> dovecot/indexer-worker [<a class="moz-txt-link-abbreviated" href="mailto:user4@email.com">user4@email.com</a>
Virtual/All](main+0xd7) [0x561e504cb1f7] ->
/lib64/libc.so.6(__libc_start_main+0xf5) [0x7f5d12e39555] ->
dovecot/indexer-worker [<a class="moz-txt-link-abbreviated" href="mailto:user4@email.com">user4@email.com</a> Virtual/All](+0x22ba)
[0x561e504cb2ba]<br>
Jun 03 05:05:34 Fatal: indexer-worker(<a class="moz-txt-link-abbreviated" href="mailto:user4@email.com">user4@email.com</a>)
session=<x0EZ2dPDiuW5Ydlx:pQBBH/tGuGDjBwAAxbYlJw>: master:
service(indexer-worker): child 2019 killed with signal 6 (core
dumped)<br>
<br>
<br>
[root@popimap ~]# gdb /usr/libexec/dovecot/indexer-worker
/var/core/core.indexer-worker.2019 <br>
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-120.el7<br>
Copyright (C) 2013 Free Software Foundation, Inc.<br>
License GPLv3+: GNU GPL version 3 or later
<a class="moz-txt-link-rfc2396E" href="http://gnu.org/licenses/gpl.html"><http://gnu.org/licenses/gpl.html></a><br>
This is free software: you are free to change and redistribute it.<br>
There is NO WARRANTY, to the extent permitted by law. Type "show
copying"<br>
and "show warranty" for details.<br>
This GDB was configured as "x86_64-redhat-linux-gnu".<br>
For bug reporting instructions, please see:<br>
<a class="moz-txt-link-rfc2396E" href="http://www.gnu.org/software/gdb/bugs/"><http://www.gnu.org/software/gdb/bugs/></a>...<br>
Reading symbols from /usr/libexec/dovecot/indexer-worker...Reading
symbols from
/usr/lib/debug/usr/libexec/dovecot/indexer-worker.debug...done.<br>
done.<br>
[New LWP 2019]<br>
[Thread debugging using libthread_db enabled]<br>
Using host libthread_db library "/lib64/libthread_db.so.1".<br>
Core was generated by `dovecot/indexer-worker'.<br>
Program terminated with signal 6, Aborted.<br>
#0 0x00007f5d12e4d387 in raise () from /lib64/libc.so.6<br>
Missing separate debuginfos, use: debuginfo-install
bzip2-libs-1.0.6-13.el7.x86_64 expat-2.1.0-12.el7.x86_64
glibc-2.17-324.el7_9.x86_64 libgcc-4.8.5-44.el7.x86_64
libzstd-1.4.9-1.el7.x86_64 xz-libs-5.2.2-1.el7.x86_64
zlib-1.2.7-19.el7_9.x86_64<br>
(gdb) bt full<br>
#0 0x00007f5d12e4d387 in raise () from /lib64/libc.so.6<br>
No symbol table info available.<br>
#1 0x00007f5d12e4ea78 in abort () from /lib64/libc.so.6<br>
No symbol table info available.<br>
#2 0x00007f5d132d8c97 in default_fatal_finish (status=0,
type=<optimized out>) at failures.c:459<br>
backtrace = 0x561e523ecab8
"/usr/lib64/dovecot/libdovecot.so.0(backtrace_append+0x42)
[0x7f5d132cbac2] ->
/usr/lib64/dovecot/libdovecot.so.0(backtrace_get+0x1e)
[0x7f5d132cbbce] ->
/usr/lib64/dovecot/libdovecot.so.0(+0xf3cde) [0"...<br>
recursed = 0<br>
#3 fatal_handler_real (ctx=<optimized out>,
format=<optimized out>, args=<optimized out>) at
failures.c:471<br>
status = 0<br>
#4 0x00007f5d132d8d81 in i_internal_fatal_handler
(ctx=<optimized out>, format=<optimized out>,
args=<optimized out>)<br>
at failures.c:866<br>
No locals.<br>
#5 0x00007f5d1322a25a in i_panic
(format=format@entry=0x7f5d1331a730 "file %s: line %d (%s):
assertion failed: (%s)")<br>
at failures.c:523<br>
ctx = {type = LOG_TYPE_PANIC, exit_status = 0, timestamp =
0x0, timestamp_usecs = 0, log_prefix = 0x0, <br>
log_prefix_type_pos = 0}<br>
args = {{gp_offset = 40, fp_offset = 48, overflow_arg_area
= 0x7ffdccb024d0, reg_save_area = 0x7ffdccb02410}}<br>
#6 0x00007f5d132729ad in http_client_request_send_more
(req=req@entry=0x561e524c2a68, pipelined=pipelined@entry=false)<br>
at http-client-request.c:1240<br>
conn = 0x561e524aa5f0<br>
cctx = 0x561e52427fa8<br>
output = 0x561e524abf60<br>
res = <optimized out><br>
error = <optimized out><br>
offset = <optimized out><br>
__func__ = "http_client_request_send_more"<br>
#7 0x00007f5d13277101 in http_client_connection_continue_request
(conn=0x561e524aa5f0) at http-client-connection.c:1342<br>
tmp_conn = 0x561e524aa5f0<br>
count = 1<br>
ret = <optimized out><br>
reqs = <optimized out><br>
req = 0x561e524c2a68<br>
pipelined = false<br>
#8 http_client_connection_output (conn=0x561e524aa5f0) at
http-client-connection.c:1374<br>
output = <optimized out><br>
ret = <optimized out><br>
#9 0x00007f5d133022b0 in stream_send_io (fstream=0x561e524ab320)
at ostream-file.c:520<br>
ostream = 0x561e524ab3c0<br>
iostream = 0x561e524ab320<br>
use_cork = true<br>
ret = <optimized out><br>
---Type <return> to continue, or q <return> to quit---<br>
#10 0x00007f5d132f0f45 in io_loop_call_io (io=0x561e524a4fb0) at
ioloop.c:714<br>
ioloop = 0x561e524c2840<br>
t_id = 6<br>
__func__ = "io_loop_call_io"<br>
#11 0x00007f5d132f28fb in io_loop_handler_run_internal
(ioloop=ioloop@entry=0x561e524c2840) at ioloop-epoll.c:222<br>
ctx = 0x561e524a4f80<br>
events = <optimized out><br>
list = 0x561e524ae350<br>
io = <optimized out><br>
tv = {tv_sec = 0, tv_usec = 0}<br>
events_count = <optimized out><br>
msecs = <optimized out><br>
ret = 1<br>
i = 0<br>
call = <optimized out><br>
__func__ = "io_loop_handler_run_internal"<br>
#12 0x00007f5d132f1049 in io_loop_handler_run
(ioloop=ioloop@entry=0x561e524c2840) at ioloop.c:766<br>
__func__ = "io_loop_handler_run"<br>
#13 0x00007f5d132f1288 in io_loop_run (ioloop=0x561e524c2840) at
ioloop.c:739<br>
__func__ = "io_loop_run"<br>
#14 0x00007f5d1328020d in http_client_wait (client=0x561e52425f98)
at http-client.c:339<br>
prev_ioloop = 0x561e523f0ff0<br>
client_ioloop = 0x561e524c2840<br>
prev_client_ioloop = 0x561e523f0ff0<br>
__func__ = "http_client_wait"<br>
#15 0x00007f5d11440174 in solr_connection_select
(conn=0x561e52425ed0, query=<optimized out>,
pool=<optimized out>, <br>
box_results_r=0x7ffdccb02780) at solr-connection.c:206<br>
lctx = {result_pool = 0x561e5249dba0, payload = 0x0, io =
0x0, request_status = 0, parser = 0x0, results = 0x0}<br>
http_req = 0x561e5249dfd8<br>
url = <optimized out><br>
#16 0x00007f5d1143c5d4 in fts_backend_solr_get_last_uid
(_backend=0x561e52425e10, box=0x561e52452538,
last_uid_r=0x7ffdccb02824)<br>
at fts-backend-solr.c:237<br>
hdr = {last_indexed_uid = 0, settings_checksum = 0, unused
= 0}<br>
#17 0x00007f5d125af3fe in fts_backend_get_last_uid
(backend=<optimized out>, box=box@entry=0x561e52452538, <br>
last_uid_r=last_uid_r@entry=0x7ffdccb02824) at fts-api.c:106<br>
hdr = {last_indexed_uid = 3434097176, settings_checksum =
32765, unused = 325539906}<br>
#18 0x00007f5d125b5952 in fts_mailbox_get_last_cached_seq
(box=box@entry=0x561e52452538, seq_r=seq_r@entry=0x7ffdccb02874)<br>
at fts-storage.c:85<br>
flist = <optimized out><br>
seq1 = 22046<br>
seq2 = 1380210400<br>
last_uid = 22046<br>
---Type <return> to continue, or q <return> to quit---<br>
#19 0x00007f5d125b6ef6 in fts_mail_precache_init
(_mail=<optimized out>, _mail=<optimized out>) at
fts-storage.c:486<br>
ft = 0x561e52499650<br>
flist = 0x561e5241e928<br>
last_seq = 22046<br>
#20 fts_mail_index (_mail=0x561e524ad188) at fts-storage.c:507<br>
ft = 0x561e52499650<br>
flist = 0x561e5241e928<br>
pmail = 0x561e524ad188<br>
#21 fts_mail_precache (_mail=0x561e524ad188) at fts-storage.c:571<br>
_data_stack_cur_id = 5<br>
mail = 0x561e524ad188<br>
fmail = <optimized out><br>
ft = 0x561e52499650<br>
__func__ = "fts_mail_precache"<br>
#22 0x00007f5d1239b66a in virtual_mail_precache
(mail=<optimized out>) at virtual-mail.c:265<br>
vmail = <optimized out><br>
backend_mail = 0x561e524ad188<br>
p = <optimized out><br>
#23 0x00007f5d125b6ba6 in fts_mail_precache (_mail=0x561e5248e0e8)
at fts-storage.c:564<br>
mail = 0x561e5248e0e8<br>
fmail = 0x561e5248e650<br>
ft = 0x561e52458670<br>
__func__ = "fts_mail_precache"<br>
#24 0x00007f5d135dbb0e in mail_precache (mail=0x561e5248e0e8) at
mail.c:453<br>
_data_stack_cur_id = 4<br>
p = 0x561e5248e0e8<br>
#25 0x0000561e504cb924 in index_mailbox_precache
(conn=<optimized out>, box=0x561e52435a18) at
master-connection.c:104<br>
counter = 1<br>
max = 172<br>
percentage_sent = 0<br>
storage = <optimized out><br>
status = {messages = 172, recent = 0, unseen = 0,
uidvalidity = 1622689531, uidnext = 173, first_unseen_seq = 0, <br>
first_recent_uid = 173, last_cached_seq = 0,
highest_modseq = 0, highest_pvt_modseq = 0, keywords = 0x0, <br>
permanent_flags = 0, flags = 0, permanent_keywords =
false, allow_new_keywords = false, nonpermanent_modseqs = false, <br>
no_modseq_tracking = false, have_guids = true,
have_save_guids = true, have_only_guid128 = false}<br>
uids = <optimized out><br>
username = 0x561e524158c8 <a class="moz-txt-link-rfc2396E" href="mailto:user4@email.com">"user4@email.com"</a><br>
first_uid = 1<br>
percentage_str = "0\000\000"<br>
percentage = <optimized out><br>
error = 1379909712<br>
trans = 0x561e524641a0<br>
---Type <return> to continue, or q <return> to quit---<br>
ctx = 0x561e52434ee0<br>
last_uid = 2<br>
ret = 0<br>
box_vname = 0x561e52435db0 "Virtual/All"<br>
errstr = <optimized out><br>
search_args = 0x0<br>
mail = 0x561e5248e0e8<br>
metadata = {guid = '\000' <repeats 15 times>,
virtual_size = 0, physical_size = 0, first_save_date = 0, <br>
cache_fields = 0x0, <br>
precache_fields = (MAIL_FETCH_STREAM_HEADER |
MAIL_FETCH_STREAM_BODY | MAIL_FETCH_RECEIVED_DATE |
MAIL_FETCH_SAVE_DATE | MAIL_FETCH_PHYSICAL_SIZE |
MAIL_FETCH_VIRTUAL_SIZE | MAIL_FETCH_UIDL_BACKEND |
MAIL_FETCH_GUID | MAIL_FETCH_POP3_ORDER), <br>
backend_ns_prefix = 0x0, backend_ns_type = (unknown: 0)}<br>
seq = <optimized out><br>
#26 index_mailbox (user=<optimized out>, user=<optimized
out>, what=<optimized out>, max_recent_msgs=<optimized
out>, <br>
mailbox=<optimized out>, conn=0x561e523fc070) at
master-connection.c:207<br>
box = 0x561e52435a18<br>
status = {messages = 0, recent = 0, unseen = 733118976,
uidvalidity = 3265295405, uidnext = 3434097280, <br>
first_unseen_seq = 32765, first_recent_uid = 1,
last_cached_seq = 0, highest_modseq = 94688228871760, <br>
highest_pvt_modseq = 140037729927678, keywords =
0x7ffdccb02a80, permanent_flags = 1379864144, flags = 22046, <br>
permanent_keywords = false, allow_new_keywords = false,
nonpermanent_modseqs = false, no_modseq_tracking = false, <br>
have_guids = true, have_save_guids = false,
have_only_guid128 = true}<br>
errstr = <optimized out><br>
ret = 0<br>
ns = <optimized out><br>
path = 0x561e523eb6d0
"/home/vmail/nas10/domains/R/email.com/user4/Maildir/virtual/All"<br>
error = 172<br>
sync_flags = <optimized out><br>
#27 master_connection_input_line (line=<optimized out>,
conn=0x561e523fc070) at master-connection.c:249<br>
str = <optimized out><br>
max_recent_msgs = 0<br>
ret = <optimized out><br>
args = <optimized out><br>
input = {parent_event = 0x0, module = 0x561e504cbfff
"mail", service = 0x561e504cbff0 "indexer-worker", <br>
username = 0x561e523e8038 <a class="moz-txt-link-rfc2396E" href="mailto:user4@email.com">"user4@email.com"</a>, session_id
= 0x0, <br>
session_id_prefix = 0x561e523e805b "x0EZ2dPDiuW5Ydlx",
session_create_time = 0, local_ip = {family = 0, u = {ip6 = {<br>
__in6_u = {__u6_addr8 = '\000' <repeats 15
times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 =
{0, 0, 0, <br>
0}}}, ip4 = {s_addr = 0}}}, remote_ip =
{family = 0, u = {ip6 = {__in6_u = {<br>
__u6_addr8 = '\000' <repeats 15 times>,
__u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0,
0}}}, <br>
ip4 = {s_addr = 0}}}, local_port = 0, remote_port =
0, userdb_fields = 0x0, forward_fields = 0x0, <br>
flags_override_add = (unknown: 0), flags_override_remove
= (unknown: 0), no_userdb_lookup = false, debug = false, <br>
conn_secured = false, conn_ssl_secured = false}<br>
service_user = 0x561e523ffe30<br>
---Type <return> to continue, or q <return> to quit---<br>
user = 0x561e524157a8<br>
error = 0x0<br>
#28 master_connection_input (conn=0x561e523fc070) at
master-connection.c:289<br>
_data_stack_cur_id = 3<br>
line = <optimized out><br>
#29 0x00007f5d132f0f45 in io_loop_call_io (io=0x561e523fc0b0) at
ioloop.c:714<br>
ioloop = 0x561e523f0ff0<br>
t_id = 2<br>
__func__ = "io_loop_call_io"<br>
#30 0x00007f5d132f28fb in io_loop_handler_run_internal
(ioloop=ioloop@entry=0x561e523f0ff0) at ioloop-epoll.c:222<br>
ctx = 0x561e523f1c30<br>
events = <optimized out><br>
list = 0x561e523fc110<br>
io = <optimized out><br>
tv = {tv_sec = 2147483, tv_usec = 0}<br>
events_count = <optimized out><br>
msecs = <optimized out><br>
ret = 1<br>
i = 0<br>
call = <optimized out><br>
__func__ = "io_loop_handler_run_internal"<br>
#31 0x00007f5d132f1049 in io_loop_handler_run
(ioloop=ioloop@entry=0x561e523f0ff0) at ioloop.c:766<br>
__func__ = "io_loop_handler_run"<br>
#32 0x00007f5d132f1288 in io_loop_run (ioloop=0x561e523f0ff0) at
ioloop.c:739<br>
__func__ = "io_loop_run"<br>
#33 0x00007f5d1325bbb3 in master_service_run
(service=0x561e523f0e50, callback=callback@entry=0x561e504cb3b0
<client_connected>)<br>
at master-service.c:853<br>
No locals.<br>
#34 0x0000561e504cb1f7 in main (argc=1, argv=0x561e523f0b90) at
indexer-worker.c:76<br>
storage_service_flags =
(MAIL_STORAGE_SERVICE_FLAG_USERDB_LOOKUP |
MAIL_STORAGE_SERVICE_FLAG_TEMP_PRIV_DROP |
MAIL_STORAGE_SERVICE_FLAG_NO_IDLE_TIMEOUT)<br>
c = <optimized out><br>
(gdb) exit<br>
</p>
<p>I hope can help to fix it.</p>
<p>Thanks<br>
</p>
<pre class="moz-signature" cols="72">--
Alessio Cecchi
Postmaster @ <a class="moz-txt-link-freetext" href="http://www.qboxmail.it">http://www.qboxmail.it</a>
<a class="moz-txt-link-freetext" href="https://www.linkedin.com/in/alessice">https://www.linkedin.com/in/alessice</a></pre>
</body>
</html>