index problems after update

Fri Feb 8 10:25:17 EET 2019

Hello List,

i have a problem with index-files which is separated in 2 subproblems. 
May be these problems are connected.
Currently we use Ubuntu 18.04 LTS which is bundled with dovecot 2.2.33.2
These servers are fresh installed machines and users are added to the 
system after, there was no upgrade.
Sometimes it happens, that dovecot stops showing new mail. There is no 
error in log, dovecot just seems to do his normal operations but is not 
delivering new emails from /var/mail/myuser
emails are delivered again if i delete index-files from .imap folder. So 
index is recreated and everything works again. I cant reproduce this 
problem, but every 2 days i have one user who is reporting this problem.

Second problem is similiar. I Upgraded a server from Ubuntu 16.04 to 
Ubuntu 18.04. This includes an upgrade from dovecot 2.2.22 to 2.2.33.2. 
After that i installed an self packaged dovecot 2.2.36.1
Now here is same problem, dovecot is not showing new mails, but the 
difference is we have a Raw backtrace in Log just as expected:

Feb  8 08:45:37 hostname dovecot[14882]: imap(myuser): Error: Raw 
backtrace: /usr/lib/dovecot/libdovecot.so.0(+0xa1ee2) [0x7f78b3b2cee2] 
-> /usr/lib/dovecot/libdovecot.so.0(+0xa1fda) [0x7f78b3b2cfda] -> 
/usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7f78b3abb8a8] -> 
/usr/lib/dovecot/libdovecot-storage.so.0(index_mail_parse_header+0x78a) 
[0x7f78b3e7940a] -> /usr/lib/dovecot/libdovecot.so.0(+0x83592) 
[0x7f78b3b0e592] -> /usr/lib/dovecot/libdovecot.so.0(i_stream_read+0x74) 
[0x7f78b3b38814] -> 
/usr/lib/dovecot/libdovecot.so.0(i_stream_read_data+0x3d) 
[0x7f78b3b392fd] -> 
/usr/lib/dovecot/libdovecot-storage.so.0(index_mail_get_header_stream+0x4a) 
[0x7f78b3e7a2ea] -> 
/usr/lib/dovecot/libdovecot-storage.so.0(mail_get_header_stream+0x4a) 
[0x7f78b3dfd8da] -> 
/usr/lib/dovecot/libdovecot-storage.so.0(imap_msgpart_open+0x24a) 
[0x7f78b3ebb57a] -> dovecot/imap [myuser ip.ip.ip.ip FETCH](+0x2081c) 
[0x5581ac76d81c] -> dovecot/imap [m038422 ip.ip.ip.ip FETCH](+0x1ea54) 
[0x5581ac76ba54] -> dovecot/imap [myuser ip.ip.ip.ip 
FETCH](imap_fetch_more+0x39) [0x5581ac76cd69] -> dovecot/imap [myuser 
ip.ip.ip.ip FETCH](cmd_fetch+0x31b) [0x5581ac75e1bb] -> dovecot/imap 
[myuser ip.ip.ip.ip FETCH](command_exec+0x5c) [0x5581ac76a09c] -> 
dovecot/imap [myuser ip.ip.ip.ip FETCH](+0x1b612) [0x5581ac768612] -> 
dovecot/imap [myuser ip.ip.ip.ip FETCH](+0x1b6ac) [0x5581ac7686ac] -> 
dovecot/imap [myuser ip.ip.ip.ip FETCH](client_handle_input+0x18d) 
[0x5581ac768a6d] -> dovecot/imap [myuser ip.ip.ip.ip 
FETCH](client_input+0xac) [0x5581ac768fbc] -> 
/usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x52) [0x7f78b3b43482] 
-> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x12e) 
[0x7f78b3b44b9e] -> 
/usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x36) 
[0x7f78b3b43516] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) 
[0x7f78b3b436c8] -> 
/usr/lib/dovecot/libdovecot.so.0(master_service_run+0x13) 
[0x7f78b3ac6aa3] -> dovecot/imap [myuser ip.ip.ip.ip FETCH](main+0x329) 
[0x5581ac75b159] -> 
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7f78b36bbb97] 
-> dovecot/imap [myuser ip.ip.ip.ip FETCH](_start+0x2a) [0x5581ac75b2fa]
Feb  8 08:45:37 hostname dovecot[14882]: imap(myuser): Fatal: master: 
service(imap): child 14135 killed with signal 6 (core dumped)

Over all dovecot and OS versions we use same old-fashioned setup. Mails 
are delivered by procmail to classic mbox file.
This is output of doveconf -n

# 2.2.36.1 (5d621cf65): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.4.21 (92477967)
# OS: Linux 4.18.0-14-generic x86_64 Ubuntu 18.04.1 LTS
# Hostname: myhostname
auth_mechanisms = plain login
auth_username_chars = 
abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_@üÜöÖäÄ
auth_worker_max_count = 60
disable_plaintext_auth = no
imap_capability = +XLIST
lock_method = flock
login_greeting = mailserver ready.
login_log_format_elements = user=<%u> method=%m rip=%r lip=%l %c
mail_cache_min_mail_count = 30
mail_location = mbox:~/mail:INBOX=/var/mail/%u
mail_max_userip_connections = 30
mbox_read_locks = dotlock
mbox_very_dirty_syncs = yes
namespace inbox {
   inbox = yes
   location =
   mailbox Archiv {
     auto = no
     special_use =
   }
   mailbox Drafts {
     auto = no
     special_use =
   }
   mailbox Junk {
     auto = no
     special_use =
   }
   mailbox Sent {
     auto = no
     special_use =
   }
   mailbox Trash {
     auto = no
     special_use =
   }
   prefix =
}
passdb {
   args = /etc/dovecot/dovecot-sql.conf
   driver = sql
}
protocols = imap pop3
service auth-worker {
   idle_kill = 1 mins
}
service auth {
   idle_kill = 1 mins
   unix_listener /var/spool/postfix/private/auth {
     group = postfix
     mode = 0660
     user = postfix
   }
   user = root
}
service imap-login {
   executable = /usr/lib/dovecot/imap-login
   process_min_avail = 4
   service_count = 0
   user = dovecot
}
service imap {
   process_limit = 1024
}
service pop3-login {
   executable = /usr/lib/dovecot/pop3-login
   process_min_avail = 4
   service_count = 0
   user = dovecot
}
service pop3 {
   executable = /usr/lib/dovecot/pop3
   process_limit = 1024
}
shutdown_clients = no
ssl_ca = </path/cert.pem
ssl_cert = </path/cert.pem
ssl_cipher_list = 
ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:!DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA
ssl_dh_parameters_length = 2048
ssl_key =  # hidden, use -P to show it
ssl_prefer_server_ciphers = yes
userdb {
   args = /etc/dovecot/dovecot-sql.conf
   driver = sql
}
verbose_proctitle = yes
protocol imap {
   imap_client_workarounds = delay-newmail
   mail_plugins = mail_log notify
   ssl = yes
}
protocol pop3 {
   pop3_client_workarounds = oe-ns-eoh outlook-no-nuls
   pop3_no_flag_updates = no
   pop3_uidl_format = %08Xu%08Xv
   ssl = yes
}

Whats going wrong here? Main problem is first one. dovecot stops showing 
new mail during service without telling a reason. reason looks like a 
broken index file.
What to do here? Please tell me your opinion.

Thanks,
Hajo