<!doctype html>
<html>
<head>
<meta charset="UTF-8">
</head>
<body>
<div>
<br>
</div>
<blockquote type="cite">
<div>
On 19 December 2018 at 20:00 Tim Mohlmann via dovecot <
<a href="mailto:dovecot@dovecot.org">dovecot@dovecot.org</a>> wrote:
</div>
<div>
<br>
</div>
<div>
<br>
</div>
<div>
Dear list,
</div>
<div>
<br>
</div>
<div>
We been having some issues where the indexer-worker is crashing. This
</div>
<div>
happens on production servers which are handling a slight amount of
</div>
<div>
mail, but is also reproducible by moving messages. Also, users on my
</div>
<div>
server are complaining about "Trashed" items coming back etc.
</div>
<div>
<br>
</div>
<div>
Some details:
</div>
<div>
<br>
</div>
<div>
- Dovecot 2.3.2.1 in alpine:3.8 based Docker container. As part of the
</div>
<div>
Mailu distribution. (
<a href="https://github.com/Mailu/Mailu" rel="noopener" target="_blank">https://github.com/Mailu/Mailu</a>)
</div>
<div>
<br>
</div>
<div>
- I've first seen this issues on my production server, which stores mail
</div>
<div>
on GlusterFS
</div>
<div>
<br>
</div>
<div>
- I've been able to reproduce running the Docker container on a Virtual
</div>
<div>
machine, using local storage.
</div>
<div>
<br>
</div>
<div>
- There is another Mailu user reporting the same problem on a different
</div>
<div>
VM provider / disk infrastructure:
<a href="https://github.com/Mailu/Mailu/issues/751" rel="noopener" target="_blank">https://github.com/Mailu/Mailu/issues/751</a>
</div>
<div>
<br>
</div>
<div>
- Libc: musl-1.1.19-r10
</div>
<div>
<br>
</div>
<div>
Output of dovecot -n:
</div>
<div>
<br>
</div>
<div>
# dovecot -n
</div>
<div>
# 2.3.2.1 (0719df592): /etc/dovecot/dovecot.conf
</div>
<div>
# Pigeonhole version 0.5.2 (7704de5e)
</div>
<div>
# OS: Linux 4.16.3-301.fc28.x86_64 x86_64 ext4
</div>
<div>
# Hostname: 98a2726271d3
</div>
<div>
auth_mechanisms = plain login
</div>
<div>
disable_plaintext_auth = no
</div>
<div>
first_valid_gid = 8
</div>
<div>
first_valid_uid = 8
</div>
<div>
hostname = mail.usrpro.io
</div>
<div>
log_path = /dev/stderr
</div>
<div>
mail_access_groups = mail
</div>
<div>
mail_gid = mail
</div>
<div>
mail_home = /mail/%u
</div>
<div>
mail_location = maildir:/mail/%u
</div>
<div>
mail_plugins = " fts fts_lucene quota quota_clone zlib"
</div>
<div>
mail_privileged_group = mail
</div>
<div>
mail_uid = mail
</div>
<div>
mail_vsize_bg_after_count = 100
</div>
<div>
maildir_stat_dirs = yes
</div>
<div>
managesieve_notify_capability = mailto
</div>
<div>
managesieve_sieve_capability = fileinto reject envelope
</div>
<div>
encoded-character vacation subaddress comparator-i;ascii-numeric
</div>
<div>
relational regex imap4flags copy include variables body enotify
</div>
<div>
environment mailbox date index ihave duplicate mime foreverypart
</div>
<div>
extracttext spamtest spamtestplus editheader imapsieve vnd.dovecot.imapsieve
</div>
<div>
namespace inbox {
</div>
<div>
inbox = yes
</div>
<div>
location =
</div>
<div>
mailbox Drafts {
</div>
<div>
auto = subscribe
</div>
<div>
special_use = \Drafts
</div>
<div>
}
</div>
<div>
mailbox Junk {
</div>
<div>
auto = subscribe
</div>
<div>
special_use = \Junk
</div>
<div>
}
</div>
<div>
mailbox Sent {
</div>
<div>
auto = subscribe
</div>
<div>
special_use = \Sent
</div>
<div>
}
</div>
<div>
mailbox Trash {
</div>
<div>
auto = subscribe
</div>
<div>
special_use = \Trash
</div>
<div>
}
</div>
<div>
prefix =
</div>
<div>
}
</div>
<div>
passdb {
</div>
<div>
args = /etc/dovecot/auth.conf
</div>
<div>
driver = dict
</div>
<div>
}
</div>
<div>
plugin {
</div>
<div>
fts = lucene
</div>
<div>
fts_autoindex = yes
</div>
<div>
fts_autoindex_exclude = \Junk
</div>
<div>
fts_lucene = whitespace_chars=@.
</div>
<div>
imapsieve_mailbox1_before = file:/conf/report-spam.sieve
</div>
<div>
imapsieve_mailbox1_causes = COPY
</div>
<div>
imapsieve_mailbox1_name = Junk
</div>
<div>
imapsieve_mailbox2_before = file:/conf/report-ham.sieve
</div>
<div>
imapsieve_mailbox2_causes = COPY
</div>
<div>
imapsieve_mailbox2_from = Junk
</div>
<div>
imapsieve_mailbox2_name = *
</div>
<div>
quota = count:User quota
</div>
<div>
quota_clone_dict = proxy:/tmp/podop.socket:quota
</div>
<div>
quota_vsizes = yes
</div>
<div>
sieve = file:~/sieve;active=~/.dovecot.sieve
</div>
<div>
sieve_before = dict:proxy:/tmp/podop.socket:sieve
</div>
<div>
sieve_execute_bin_dir = /conf/bin
</div>
<div>
sieve_extensions = +spamtest +spamtestplus +editheader
</div>
<div>
sieve_global_extensions = +vnd.dovecot.execute
</div>
<div>
sieve_plugins = sieve_imapsieve sieve_extprograms
</div>
<div>
sieve_spamtest_max_value = 15
</div>
<div>
sieve_spamtest_status_header = X-Spam-Level
</div>
<div>
sieve_spamtest_status_type = strlen
</div>
<div>
sieve_vacation_dont_check_recipient = yes
</div>
<div>
sieve_vacation_send_from_recipient = yes
</div>
<div>
}
</div>
<div>
postmaster_address =
<a href="mailto:admin@usrpro.io">admin@usrpro.io</a>
</div>
<div>
protocols = imap pop3 lmtp sieve
</div>
<div>
service auth-worker {
</div>
<div>
unix_listener auth-worker {
</div>
<div>
group = mail
</div>
<div>
mode = 0660
</div>
<div>
user = dovecot
</div>
<div>
}
</div>
<div>
user = mail
</div>
<div>
}
</div>
<div>
service auth {
</div>
<div>
user = dovecot
</div>
<div>
}
</div>
<div>
service imap-login {
</div>
<div>
inet_listener imap {
</div>
<div>
port = 143
</div>
<div>
}
</div>
<div>
}
</div>
<div>
service lmtp {
</div>
<div>
inet_listener lmtp {
</div>
<div>
port = 2525
</div>
<div>
}
</div>
<div>
}
</div>
<div>
service managesieve-login {
</div>
<div>
inet_listener sieve {
</div>
<div>
port = 4190
</div>
<div>
}
</div>
<div>
}
</div>
<div>
submission_host = 192.168.203.6
</div>
<div>
userdb {
</div>
<div>
args = /etc/dovecot/auth.conf
</div>
<div>
driver = dict
</div>
<div>
}
</div>
<div>
protocol imap {
</div>
<div>
mail_plugins = " fts fts_lucene quota quota_clone zlib imap_quota
</div>
<div>
imap_sieve"
</div>
<div>
}
</div>
<div>
protocol lmtp {
</div>
<div>
mail_plugins = " fts fts_lucene quota quota_clone zlib sieve"
</div>
<div>
recipient_delimiter = +
</div>
<div>
}
</div>
<div>
<br>
</div>
<div>
And the actual error log:
</div>
<div>
<br>
</div>
<div>
imap_1 | Dec 19 16:31:08
</div>
<div>
indexer-worker(
<a href="mailto:admin@usrpro.io">admin@usrpro.io</a>)<490><m+t5VmJ93K7AqMsG:grc9HUxyGlzqAQAANEhNiw>:
</div>
<div>
Fatal: master: service(indexer-worker): child 490 killed with signal 11
</div>
<div>
(core dumped)
</div>
<div>
imap_1 | Dec 19 16:31:09
</div>
<div>
indexer-worker(
<a href="mailto:admin@usrpro.io">admin@usrpro.io</a>)<493><m+t5VmJ93K7AqMsG:HRLEK0xyGlztAQAANEhNiw>:
</div>
<div>
Error: lucene index
<a href="mailto:/mail/admin@usrpro.io">/mail/admin@usrpro.io</a>/lucene-indexes: IndexWriter()
</div>
<div>
failed (#1): Lock obtain timed out
</div>
<div>
imap_1 | Dec 19 16:31:09
</div>
<div>
indexer-worker(
<a href="mailto:admin@usrpro.io">admin@usrpro.io</a>)<493><m+t5VmJ93K7AqMsG:HRLEK0xyGlztAQAANEhNiw>:
</div>
<div>
Error: Mailbox INBOX: Mail search failed: Internal error occurred. Refer
</div>
<div>
to server log for more information. [2018-12-19 16:31:08]
</div>
<div>
imap_1 | Dec 19 16:31:09
</div>
<div>
indexer-worker(
<a href="mailto:admin@usrpro.io">admin@usrpro.io</a>)<493><m+t5VmJ93K7AqMsG:HRLEK0xyGlztAQAANEhNiw>:
</div>
<div>
Error: Mailbox INBOX: Transaction commit failed: FTS transaction commit
</div>
<div>
failed: backend deinit (attempted to index 1 messages (UIDs 1299..1299))
</div>
<div>
imap_1 | Dec 19 16:31:10
</div>
<div>
indexer-worker(
<a href="mailto:admin@usrpro.io">admin@usrpro.io</a>)<493><m+t5VmJ93K7AqMsG:GKWdMU1yGlztAQAANEhNiw>:
</div>
<div>
Fatal: master: service(indexer-worker): child 493 killed with signal 11
</div>
<div>
(core dumped)
</div>
<div>
imap_1 | Dec 19 16:31:10 indexer: Error: Indexer worker
</div>
<div>
disconnected, discarding 1 requests for
<a href="mailto:admin@usrpro.io">admin@usrpro.io</a>
</div>
<div>
imap_1 | Dec 19 16:31:11
</div>
<div>
indexer-worker(
<a href="mailto:admin@usrpro.io">admin@usrpro.io</a>)<494><m+t5VmJ93K7AqMsG:MRCzBE5yGlzuAQAANEhNiw>:
</div>
<div>
Error: lucene index
<a href="mailto:/mail/admin@usrpro.io">/mail/admin@usrpro.io</a>/lucene-indexes: IndexWriter()
</div>
<div>
failed (#1): Lock obtain timed out
</div>
<div>
imap_1 | Dec 19 16:31:11
</div>
<div>
indexer-worker(
<a href="mailto:admin@usrpro.io">admin@usrpro.io</a>)<494><m+t5VmJ93K7AqMsG:MRCzBE5yGlzuAQAANEhNiw>:
</div>
<div>
Error: Mailbox INBOX: Mail search failed: Internal error occurred. Refer
</div>
<div>
to server log for more information. [2018-12-19 16:31:10]
</div>
<div>
imap_1 | Dec 19 16:31:11
</div>
<div>
indexer-worker(
<a href="mailto:admin@usrpro.io">admin@usrpro.io</a>)<494><m+t5VmJ93K7AqMsG:MRCzBE5yGlzuAQAANEhNiw>:
</div>
<div>
Error: Mailbox INBOX: Transaction commit failed: FTS transaction commit
</div>
<div>
failed: backend deinit (attempted to index 1 messages (UIDs 1310..1310))
</div>
<div>
imap_1 | Dec 19 16:31:11 indexer: Error: Indexer worker
</div>
<div>
disconnected, discarding 1 requests for
<a href="mailto:admin@usrpro.io">admin@usrpro.io</a>
</div>
<div>
<br>
</div>
<div>
I managed to find a core dump file, which appeared outside of the
</div>
<div>
container. So I copied it back in, installed and ran gdb:
</div>
<div>
<br>
</div>
<div>
GNU gdb (GDB) 8.0.1
</div>
<div>
Copyright (C) 2017 Free Software Foundation, Inc.
</div>
<div>
License GPLv3+: GNU GPL version 3 or later
</div>
<div>
<
<a href="http://gnu.org/licenses/gpl.html" rel="noopener" target="_blank">http://gnu.org/licenses/gpl.html</a>>
</div>
<div>
This is free software: you are free to change and redistribute it.
</div>
<div>
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
</div>
<div>
and "show warranty" for details.
</div>
<div>
This GDB was configured as "x86_64-alpine-linux-musl".
</div>
<div>
Type "show configuration" for configuration details.
</div>
<div>
For bug reporting instructions, please see:
</div>
<div>
<
<a href="http://www.gnu.org/software/gdb/bugs/>" rel="noopener" target="_blank" data-mce-href="http://www.gnu.org/software/gdb/bugs/">">http://www.gnu.org/software/gdb/bugs/></a>.
</div>
<div>
Find the GDB manual and other documentation resources online at:
</div>
<div>
<
<a href="http://www.gnu.org/software/gdb/documentation/>" rel="noopener" target="_blank" data-mce-href="http://www.gnu.org/software/gdb/documentation/">">http://www.gnu.org/software/gdb/documentation/></a>.
</div>
<div>
For help, type "help".
</div>
<div>
Type "apropos word" to search for commands related to "word"...
</div>
<div>
Reading symbols from /usr/libexec/dovecot/indexer-worker...(no debugging
</div>
<div>
symbols found)...done.
</div>
<div>
[New LWP 1075]
</div>
<div>
<br>
</div>
<div>
warning: Can't read pathname for load map: No error information.
</div>
<div>
Core was generated by `dovecot/indexer-worker'.
</div>
<div>
Program terminated with signal SIGSEGV, Segmentation fault.
</div>
<div>
#0 0x00007fbd9a31c11a in free () from /lib/ld-musl-x86_64.so.1
</div>
<div>
<br>
</div>
<div>
So this seems musl related. I installed musl-dbg and ran again:
</div>
<div>
<br>
</div>
<div>
GNU gdb (GDB) 8.0.1
</div>
<div>
Copyright (C) 2017 Free Software Foundation, Inc.
</div>
<div>
License GPLv3+: GNU GPL version 3 or later
</div>
<div>
<
<a href="http://gnu.org/licenses/gpl.html" rel="noopener" target="_blank">http://gnu.org/licenses/gpl.html</a>>
</div>
<div>
This is free software: you are free to change and redistribute it.
</div>
<div>
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
</div>
<div>
and "show warranty" for details.
</div>
<div>
This GDB was configured as "x86_64-alpine-linux-musl".
</div>
<div>
Type "show configuration" for configuration details.
</div>
<div>
For bug reporting instructions, please see:
</div>
<div>
<
<a href="http://www.gnu.org/software/gdb/bugs/>" rel="noopener" target="_blank" data-mce-href="http://www.gnu.org/software/gdb/bugs/">">http://www.gnu.org/software/gdb/bugs/></a>.
</div>
<div>
Find the GDB manual and other documentation resources online at:
</div>
<div>
<
<a href="http://www.gnu.org/software/gdb/documentation/>" rel="noopener" target="_blank" data-mce-href="http://www.gnu.org/software/gdb/documentation/">">http://www.gnu.org/software/gdb/documentation/></a>.
</div>
<div>
For help, type "help".
</div>
<div>
Type "apropos word" to search for commands related to "word"...
</div>
<div>
Reading symbols from /usr/libexec/dovecot/indexer-worker...(no debugging
</div>
<div>
symbols found)...done.
</div>
<div>
[New LWP 1075]
</div>
<div>
<br>
</div>
<div>
warning: Can't read pathname for load map: No error information.
</div>
<div>
Core was generated by `dovecot/indexer-worker'.
</div>
<div>
Program terminated with signal SIGSEGV, Segmentation fault.
</div>
<div>
#0 a_crash () at ./arch/x86_64/atomic_arch.h:108
</div>
<div>
108 ./arch/x86_64/atomic_arch.h: No such file or directory.
</div>
<div>
<br>
</div>
<div>
Now I'm kinda lost in space. I don't know where that header file is.
</div>
<div>
Tried running "find" on the filesystem and a google search. But nothing
</div>
<div>
specific showed up.
</div>
<div>
<br>
</div>
<div>
I am starting to feel this bug is more musl related than Dovecot. Since
</div>
<div>
this has bitten our project more in the past, I'm considering to move
</div>
<div>
the project to Debian based images. But I want to be 100% sure this is
</div>
<div>
not a dovecot bug.
</div>
<div>
<br>
</div>
<div>
Note, earlier I created an image with Alpine:edge with musl 1.1.20-r2
</div>
<div>
and Dovecot 2.3.3. Running that image I saw the same error symptoms (Was
</div>
<div>
just a hopeful trail and error). I did not do any debugging on that one.
</div>
<div>
<br>
</div>
<div>
Thanks in advance! Tim
</div>
<div>
<br>
</div>
<div>
<br>
</div>
</blockquote>
<div>
Can you run bt full on gdb and post that?
</div>
<div class="io-ox-signature">
---
<br>Aki Tuomi
</div>
</body>
</html>