Re: Bug in dovecot 2.3 virtual plugin

5 Feb 2018


      Yeah, sorry I forgot to report back, it's now fixed in
https://github.com/dovecot/core/commit/5c1837529e6957da3e389683c43bd00685939...
Aki
On 05.02.2018 14:04, Marco Giunta wrote:
...
Hi,
did you have time to investigate about 'Panic: file unichar.c' bug ?
Because I have the same problem with a 2.3.0 installation without
virtual plugin.
Thanks,
Marco
On 2018-01-03 16:52, Aki Tuomi wrote:
...
This is not a bug in virtual plugin, but in some email which contains
invalid unicode sequence somehow. Can you send me a core file? This
should not have occured ofc but would be nice to know how it ended up
here.
Aki
...
On January 3, 2018 at 5:35 PM Jakobus Schürz
<wertstoffe@nurfuerspam.de> wrote:
Hi there!
I compiled dovecot 2.3 from git. Because there is already a bug in
virtual-plugin, and i hoped, it get fixed... but it doesn't. So this
is the error-message from the log
Jän 03 16:27:08 aldebaran dovecot[26460]:
indexer-worker(jakob)<26476><qQ6g1+BhIJvAqAAO:sjYhMTH2TFpsZwAAk1Mx3g>:
Panic: file unichar.c: line 160 (uni_ucs4_to_utf8_c): assertion
failed: (uni_is_valid_ucs4(chr))
Jän 03 16:27:08 aldebaran dovecot[26460]:
indexer-worker(jakob)<26476><qQ6g1+BhIJvAqAAO:sjYhMTH2TFpsZwAAk1Mx3g>:
Error: Raw backtrace: /usr/local/lib/dovecot/libdovecot.so.0(+0xc6021)
[0x7f8299f7a021] -> /usr/local/lib/dovecot/libdovecot.so.0(+0xc60ed)
[0x7f8299f7a0ed] -> /usr/local/lib/dovecot/libdovecot.so.0(i_fatal+0)
[0x7f8299eec481] ->
/usr/local/lib/dovecot/libdovecot.so.0(uni_ucs4_to_utf8_c+0xa0)
[0x7f8299fb1500] -> /usr/local/lib/dovecot/libdovecot.so.0(+0xa75e0)
[0x7f8299f5b5e0] ->
/usr/local/lib/dovecot/libdovecot.so.0(mail_html2text_more+0xc5)
[0x7f8299f5b775] ->
/usr/local/lib/dovecot/lib20_fts_plugin.so(+0xcfcc) [0x7f82990aefcc]
-> /usr/local/lib/dovecot/lib20_fts_plugin.so(fts_parser_more+0x27)
[0x7f82990aeca7] ->
/usr/local/lib/dovecot/lib20_fts_plugin.so(fts_build_mail+0x5e9)
[0x7f82990acc39] ->
/usr/local/lib/dovecot/lib20_fts_plugin.so(+0x1122d) [0x7f82990b322d]
-> /usr/local/lib/dovecot/lib20_virtual_plugin.so(+0x916a)
[0x7f82958e316a] ->
/usr/local/lib/dovecot/lib20_fts_plugin.so(+0x10f5d) [0x7f82990b2f5d]
-> /usr/local/lib/dovecot/lib20_virtual_plugin.so(+0x916a)
[0x7f82958e316a] ->
/usr/local/lib/dovecot/lib20_fts_plugin.so(+0x10f5d) [0x7f82990b2f5d]
-> /usr/local/lib/dovecot/libdovecot-storage.so.0(mail_precache+0x2e)
[0x7f829a2641be] -> dovecot/indexer-worker jakob
Synoptic/AKTUELL [0x562227882533] ->
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x69)
[0x7f8299f91bf9] ->
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x109)
[0x7f8299f93499] ->
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x52)
[0x7f8299f91d02] ->
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_run+0x38)
[0x7f8299f91f18] ->
/usr/local/lib/dovecot/libdovecot.so.0(master_service_run+0x13)
[0x7f8299f0f1e3] -> dovecot/indexer-worker jakob
Synoptic/AKTUELL [0x562227881f47] ->
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1)
[0x7f8299b352b1] -> dovecot/indexer-worker jakob
Synoptic/AKTUELL [0x56222788201a]
Jän 03 16:27:08 aldebaran dovecot[26460]: indexer: Error: Indexer
worker disconnected, discarding 1 requests for jakob
Jän 03 16:27:08 aldebaran dovecot[26460]:
imap(jakob)<26472><qQ6g1+BhIJvAqAAO>: Error: indexer failed to index
mailbox Synoptic/AKTUELL
Jän 03 16:27:08 aldebaran dovecot[26460]:
indexer-worker(jakob)<26476><qQ6g1+BhIJvAqAAO:sjYhMTH2TFpsZwAAk1Mx3g>:
Fatal: master: service(indexer-worker): child 26476 killed with signal
6 (core dumps disabled)
Jän 03 16:27:09 aldebaran dovecot[26460]:
indexer-worker(jakob)<26484><qQ6g1+BhIJvAqAAO:J6mUIEz2TFp0ZwAAk1Mx3g>:
Error: lucene index
/var/lib/dovecot/db/indexes/Maildir/jakob/lucene-indexes:
IndexWriter() failed (#1): Lock obtain timed out
Jän 03 16:27:10 aldebaran dovecot[26460]:
indexer-worker(jakob)<26484><qQ6g1+BhIJvAqAAO:J6mUIEz2TFp0ZwAAk1Mx3g>:
Error: Mailbox Synoptic/AKTUELL: Transaction commit failed: BUG:
Unknown internal error (attempted to index 1488 messages (UIDs
15214..16775))
Jän 03 16:27:10 aldebaran dovecot[26460]:
imap(jakob)<26480><lP5r2OBhKJvAqAAO>: Error: indexer failed to index
mailbox Synoptic/AKTUELL
Jän 03 16:27:11 aldebaran dovecot[26460]:
imap(jakob)<26472><qQ6g1+BhIJvAqAAO>: Panic: file mail-index.c: line
793 (mail_index_close): assertion failed: (index->open_count > 0)
Jän 03 16:27:11 aldebaran dovecot[26460]:
imap(jakob)<26472><qQ6g1+BhIJvAqAAO>: Error: Raw backtrace:
/usr/local/lib/dovecot/libdovecot.so.0(+0xc6021) [0x7fb0fbd3a021] ->
/usr/local/lib/dovecot/libdovecot.so.0(+0xc60ed) [0x7fb0fbd3a0ed] ->
/usr/local/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7fb0fbcac481] ->
/usr/local/lib/dovecot/libdovecot-storage.so.0(+0xf39a8)
[0x7fb0fc0d99a8] ->
/usr/local/lib/dovecot/libdovecot-storage.so.0(+0xd92d1)
[0x7fb0fc0bf2d1] ->
/usr/local/lib/dovecot/libdovecot-storage.so.0(+0xd9363)
[0x7fb0fc0bf363] ->
/usr/local/lib/dovecot/libdovecot-storage.so.0(index_storage_mailbox_free+0x9)
[0x7fb0fc0b0f39] ->
/usr/local/lib/dovecot/libdovecot-storage.so.0(+0xd05a4)
[0x7fb0fc0b65a4] ->
/usr/local/lib/dovecot/libdovecot-storage.so.0(mailbox_free+0x19)
[0x7fb0fc030c99] ->
/usr/local/lib/dovecot/lib20_virtual_plugin.so(+0xabdc)
[0x7fb0f729bbdc] ->
/usr/local/lib/dovecot/lib20_virtual_plugin.so(+0xacb9)
[0x7fb0f729bcb9] ->
/usr/local/lib/dovecot/libdovecot-storage.so.0(mailbox_close+0x1a)
[0x7fb0fc030c0a] ->
/usr/local/lib/dovecot/lib20_virtual_plugin.so(+0xac60)
[0x7fb0f729bc60] ->
/usr/local/lib/dovecot/lib20_virtual_plugin.so(+0xacb9)
[0x7fb0f729bcb9] ->
/usr/local/lib/dovecot/libdovecot-storage.so.0(mailbox_close+0x1a)
[0x7fb0fc030c0a] ->
/usr/local/lib/dovecot/libdovecot-storage.so.0(mailbox_free+0x13)
[0x7fb0fc030c93] -> dovecot/imap jakob 192.168.0.14
LOGOUT [0x56514d31305b] ->
dovecot/imap jakob 192.168.0.14 LOGOUT
[0x56514d309485] -> dovecot/imap jakob 192.168.0.14
LOGOUT [0x56514d312ca4] -> dovecot/imap jakob
192.168.0.14 LOGOUT [0x56514d3110b2] -> dovecot/imap jakob
192.168.0.14 LOGOUT [0x56514d31113c] -> dovecot/imap jakob
192.168.0.14 LOGOUT [0x56514d311565] ->
dovecot/imap jakob 192.168.0.14 LOGOUT
[0x56514d311be2] ->
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x69)
[0x7fb0fbd51bf9] ->
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x109)
[0x7fb0fbd53499] ->
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x52)
[0x7fb0fbd51d02] ->
/usr/local/lib/dovecot/libdovecot.so.0(io_loop_run+0x38)
[0x7fb0fbd51f18] ->
/usr/local/lib/dovecot/libdovecot.so.0(master_service_run+0x13)
[0x7fb0fbccf1e3]
Jän 03 16:27:11 aldebaran dovecot[26460]:
imap(jakob)<26472><qQ6g1+BhIJvAqAAO>: Fatal: master: service(imap):
child 26472 killed with signal 6 (core dumps disabled)
I get this errors when i open or search in a virtual folder. i have
enabled fts_lucene.
My config is:
# dovecot -n
# 2.3.devel (393bfbea8): /usr/local/etc/dovecot/dovecot.conf
# Pigeonhole version 0.5.0.rc1 (d68c23a1)
# OS: Linux 4.13.0-trunk-amd64 x86_64 Debian 9.3 btrfs
auth_mechanisms = plain login cram-md5 digest-md5
auth_socket_path = /usr/local/var/run/dovecot/auth-userdb
first_valid_uid = 1000
imap_capability = +XDOVECOT
imap_client_workarounds = tb-extra-mailbox-sep
lda_mailbox_autocreate = yes
lda_mailbox_autosubscribe = yes
listen = *, ::1
login_trusted_networks = 127.0.0.1/8 192.168.0.0/24 192.168.1.0/24
172.17.0.0/24 172.18.0.0/24
mail_gid = vmail
mail_home =  /var/mail/%u
mail_location =
maildir:/var/mail/%u/Maildir:LAYOUT=fs:INDEX=/var/lib/dovecot/db/indexes/Maildir/%u:INDEXPVT=/var/lib/dovecot/db/indexes/Maildir/%u:CONTROL=/var/lib/dovecot/db/control/Maildir/%u
mail_plugins = zlib quota acl listescape mail_log notify virtual fts
fts_lucene
mail_privileged_group = vmail
mail_server_admin = mailto:jakob@xundeenergie.at
mail_shared_explicit_inbox = yes
mail_uid = vmail
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope
encoded-character vacation subaddress comparator-i;ascii-numeric
relational regex imap4flags copy include variables body enotify
environment mailbox date index ihave duplicate mime foreverypart
extracttext
namespace {
list = children
location =
maildir:/var/mail/public/:LAYOUT=fs:INDEX=/var/lib/dovecot/db/indexes/public/%u:INDEXPVT=/var/lib/dovecot/db/indexes/public/%u
    prefix = Roseggergasse/
separator = /
subscriptions = no
type = public
}
namespace Geteilt {
hidden = no
inbox = no
list = children
location =
maildir:/var/mail/%%u/Maildir:LAYOUT=fs:INDEXPVT=/var/lib/dovecot/db/indexes/shared/%u/%%u:INDEX=/var/lib/dovecot/db/indexes/shared/%u/%%u
    prefix = Geteilt/%%n/
separator = /
subscriptions = no
type = shared
}
namespace Mailarchiv {
hidden = no
inbox = no
list = children
location =
maildir:/var/mail/mailarchiv/%u/:LAYOUT=fs:INDEX=/var/lib/dovecot/db/indexes/mailarchiv/%u:INDEXPVT=/var/lib/dovecot/db/indexes/mailarchiv/%u
    mailbox incoming {
auto = create
}
mailbox outgoing {
auto = create
}
prefix = Mailarchiv/
separator = /
subscriptions = no
type = private
}
namespace Real {
hidden = yes
list = no
location =
virtual:/var/mail/real:INDEX=/var/lib/dovecot/db/indexes/real/%u
prefix = Real/
separator = /
subscriptions = no
}
namespace Synoptic {
hidden = no
list = children
location =
virtual:/var/mail/virtual:INDEX=/var/lib/dovecot/db/indexes/virtual/%u
mailbox INBOX/Archives {
auto = no
special_use = \Archive
}
mailbox INBOX/Drafts {
auto = no
special_use = \Drafts
}
mailbox INBOX/Entwürfe {
auto = no
special_use = \Drafts
}
mailbox INBOX/Junk {
auto = no
special_use = \Junk
}
mailbox INBOX/Sent {
auto = no
special_use = \Sent
}
mailbox INBOX/Spam {
auto = no
special_use = \Junk
}
prefix = Synoptic/
separator = /
subscriptions = no
}
namespace inbox {
hidden = no
inbox = yes
location =
mailbox Archiv {
auto = no
special_use = \Archive
}
mailbox Archive {
auto = no
special_use = \Archive
}
mailbox Archives {
auto = no
special_use = \Archive
}
mailbox "Deleted Messages" {
auto = no
autoexpunge = 30 days
special_use = \Trash
}
mailbox Drafts {
auto = no
special_use = \Drafts
}
mailbox Entwürfe {
auto = no
special_use = \Drafts
}
mailbox "Gelöschte Elemente" {
auto = no
autoexpunge = 30 days
special_use = \Trash
}
mailbox "Gelöschte Objekte" {
auto = no
autoexpunge = 30 days
special_use = \Trash
}
mailbox Gesendet {
auto = no
special_use = \Sent
}
mailbox "Gesendete Elemente" {
auto = no
special_use = \Sent
}
mailbox "Gesendete Objekte" {
auto = no
special_use = \Sent
}
mailbox Important {
auto = no
}
mailbox Junk {
auto = subscribe
autoexpunge = 30 days
special_use = \Junk
}
mailbox Mistkübel {
auto = no
autoexpunge = 30 days
special_use = \Trash
}
mailbox Papierkorb {
auto = no
autoexpunge = 30 days
special_use = \Trash
}
mailbox Sent {
auto = subscribe
special_use = \Sent
}
mailbox "Sent Messages" {
auto = no
special_use = \Sent
}
mailbox Spam {
auto = no
autoexpunge = 30 days
special_use = \Junk
}
mailbox Synoptic/Alle {
auto = no
comment = All my messages
special_use = \All
}
mailbox Trash {
auto = no
autoexpunge = 30 days
special_use = \Trash
}
mailbox Wichtig {
auto = create
}
prefix = INBOX/
separator = /
subscriptions = no
type = private
}
namespace subscriptions {
hidden = yes
list = no
location =
prefix =
subscriptions = yes
}
passdb {
args = scheme=CRYPT username_format=%u /usr/local/etc/dovecot/users
driver = passwd-file
}
plugin {
acl = vfile:/etc/dovecot/dovecot-acl:cache_secs=300
acl_shared_dict = file:/var/lib/dovecot/db/shared-mailboxes.db
fts = lucene
fts_autoindex = yes
fts_autoindex_max_recent_msgs = 80
fts_index_timeout = 90s
fts_lucene = whitespace_chars=@. normalize no_snowball
mail_home =  /var/mail/%u
setting_name = sieve, managedsieve
sieve = file:/var/mail/%u/sieve/;active=/var/mail/%u/sieve/%u.sieve
}
postmaster_address = postmaster@localhost
protocols = imap pop3 lmtp imap lmtp sieve pop3 sieve
service anvil {
unix_listener anvil-auth-penalty {
mode = 00
}
}
service auth {
unix_listener auth-client {
group = vmail
mode = 0666
user = Debian-exim
}
unix_listener auth-userdb {
group = vmail
mode = 0666
user = vmail
}
}
service imap-login {
inet_listener imap {
port = 143
}
inet_listener imaps {
port = 993
ssl = yes
}
process_min_avail = 1
service_count = 0
}
service imap {
executable = imap postlogin
process_limit = 1024
vsz_limit = 400 M
}
service lmtp {
executable = lmtp -L
user = vmail
vsz_limit = 400 M
}
service postlogin {
executable = script-login -d rawlog
}
ssl = required
ssl_cert = </usr/local/etc/dovecot/dovecot.crt
ssl_dh =  # hidden, use -P to show it
ssl_key =  # hidden, use -P to show it
userdb {
args = username_format=%u /usr/local/etc/dovecot/users
default_fields = home=/var/mail/%u
driver = passwd-file
}
verbose_proctitle = yes
protocol lmtp {
auth_username_format = %n
mail_plugins = zlib quota acl listescape mail_log notify virtual
fts fts_lucene quota sieve acl
}
protocol lda {
mail_plugins = zlib quota acl listescape mail_log notify virtual
fts fts_lucene quota sieve acl
}
protocol imap {
mail_max_userip_connections = 10
mail_plugins = zlib quota acl listescape mail_log notify virtual
fts fts_lucene imap_quota imap_acl
}
The virtual Folder-definitions are:
# find /var/mail/virtual/ /var/mail/real/
/var/mail/virtual/
/var/mail/virtual/GINBOX
/var/mail/virtual/GINBOX/dovecot-virtual
/var/mail/virtual/Neue
/var/mail/virtual/Neue/dovecot-virtual
/var/mail/virtual/Markierte
/var/mail/virtual/Markierte/dovecot-virtual
/var/mail/virtual/Ungelesene
/var/mail/virtual/Ungelesene/dovecot-virtual
/var/mail/virtual/Sent
/var/mail/virtual/Sent/dovecot-virtual
/var/mail/virtual/Drafts
/var/mail/virtual/Drafts/dovecot-virtual
/var/mail/virtual/Trash
/var/mail/virtual/Trash/dovecot-virtual
/var/mail/virtual/AKTUELL
/var/mail/virtual/AKTUELL/dovecot-virtual
/var/mail/virtual/Archives
/var/mail/virtual/Archives/dovecot-virtual
/var/mail/virtual/dovecot-acl-list
/var/mail/virtual/WICHTIG
/var/mail/virtual/WICHTIG/dovecot-virtual
/var/mail/real/
/var/mail/real/Alle
/var/mail/real/Alle/dovecot-virtual
/var/mail/real/dovecot-acl-list
/var/mail/real/einjahr
/var/mail/real/einjahr/dovecot-virtual
/var/mail/real/zweijahr
/var/mail/real/zweijahr/dovecot-virtual
/var/mail/real/dreijahr
/var/mail/real/dreijahr/dovecot-virtual
# cat /var/mail/virtual/AKTUELL/dovecot-virtual
Real/Alle
inthread refs younger 2592000
   # cat /var/mail/real/Alle/dovecot-virtual
!INBOX
INBOX/*
-INBOX/Trash
-INBOX/Trash/*
-INBOX/Archives
-INBOX/Archives/*
-INBOX/Drafts
-INBOX/Drafts/*
-INBOX/Junk
-INBOX/Junk/*
-INBOX/SYSTEM
-INBOX/SYSTEM/*
all
The others are similar.
Greez
Jakob