index problems after update

Fri Feb 15 09:47:19 EET 2019

Hello,

Am 11.02.2019 um 09:12 schrieb Hajo Locke via dovecot:
> Hello,
> Am 08.02.2019 um 09:25 schrieb Hajo Locke via dovecot:
>> Hello List,
>>
>> i have a problem with index-files which is separated in 2 
>> subproblems. May be these problems are connected.
>> Currently we use Ubuntu 18.04 LTS which is bundled with dovecot 2.2.33.2
>> These servers are fresh installed machines and users are added to the 
>> system after, there was no upgrade.
>> Sometimes it happens, that dovecot stops showing new mail. There is 
>> no error in log, dovecot just seems to do his normal operations but 
>> is not delivering new emails from /var/mail/myuser
>> emails are delivered again if i delete index-files from .imap folder. 
>> So index is recreated and everything works again. I cant reproduce 
>> this problem, but every 2 days i have one user who is reporting this 
>> problem.
>>

Iam surprised that no one has an opinion in this case. Nobody else 
noticed this problem?
Yesterday i had a customer who was affected twice. First time we saw an 
errormessage, which leads to an index problem:

Feb 13 15:46:58 hostname dovecot[30097]: imap(username): Error: 
Corrupted record in index cache file 
/home/popuser/username/mail/.imap/INBOX/dovecot.index.cache: UID 26876: 
Broken physical size in mailbox INBOX: read(/var/mail/username) failed: 
Cached message size smaller than expected (2118 < 8088, box=INBOX, 
UID=26876)
Feb 13 15:46:58 hostname dovecot[30097]: imap(username): Error: 
read(/var/mail/username) failed: Cached message size smaller than 
expected (2118 < 8088, box=INBOX, UID=26876) (FETCH BODY[] for mailbox 
INBOX UID 26876)

At this point no new incoming mail is delivered by imapd. We deleted 
index files, dovecot reacreated them and all was working again.
a day later same user was affected again. But like all other cases 
before there is no error or unusual logline. i turned mail_debug on, but 
no further hint.
There are definitely new Mails in /var/mail/username but dovecot still 
shows old state from cache. I deleted index-files again and imap-client 
showed a lot of new unread mails.
In log i can see that people are active in imap-clients and managing 
mails. If a new mail is coming in this period it seems to harm 
index-files some times. We never noticed same behaviour with long used 
dovecot 2.2.22

It seems that 2.2.33.2 which is bundled with Ubuntu18.04 LTS has a 
special problem.

Please help. What is your advice?

>> Second problem is similiar. I Upgraded a server from Ubuntu 16.04 to 
>> Ubuntu 18.04. This includes an upgrade from dovecot 2.2.22 to 
>> 2.2.33.2. After that i installed an self packaged dovecot 2.2.36.1
>> Now here is same problem, dovecot is not showing new mails, but the 
>> difference is we have a Raw backtrace in Log just as expected:
>>
> i think this crash is a special problem of dovecot 2.2.36.1.
> if i downgrade to 2.2.33.2 all is working well, upgrading again to 
> 2.2.36.1 leeds to same crash.
>> Feb  8 08:45:37 hostname dovecot[14882]: imap(myuser): Error: Raw 
>> backtrace: /usr/lib/dovecot/libdovecot.so.0(+0xa1ee2) 
>> [0x7f78b3b2cee2] -> /usr/lib/dovecot/libdovecot.so.0(+0xa1fda) 
>> [0x7f78b3b2cfda] -> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) 
>> [0x7f78b3abb8a8] -> 
>> /usr/lib/dovecot/libdovecot-storage.so.0(index_mail_parse_header+0x78a) 
>> [0x7f78b3e7940a] -> /usr/lib/dovecot/libdovecot.so.0(+0x83592) 
>> [0x7f78b3b0e592] -> 
>> /usr/lib/dovecot/libdovecot.so.0(i_stream_read+0x74) [0x7f78b3b38814] 
>> -> /usr/lib/dovecot/libdovecot.so.0(i_stream_read_data+0x3d) 
>> [0x7f78b3b392fd] -> 
>> /usr/lib/dovecot/libdovecot-storage.so.0(index_mail_get_header_stream+0x4a) 
>> [0x7f78b3e7a2ea] -> 
>> /usr/lib/dovecot/libdovecot-storage.so.0(mail_get_header_stream+0x4a) 
>> [0x7f78b3dfd8da] -> 
>> /usr/lib/dovecot/libdovecot-storage.so.0(imap_msgpart_open+0x24a) 
>> [0x7f78b3ebb57a] -> dovecot/imap [myuser ip.ip.ip.ip FETCH](+0x2081c) 
>> [0x5581ac76d81c] -> dovecot/imap [m038422 ip.ip.ip.ip 
>> FETCH](+0x1ea54) [0x5581ac76ba54] -> dovecot/imap [myuser ip.ip.ip.ip 
>> FETCH](imap_fetch_more+0x39) [0x5581ac76cd69] -> dovecot/imap [myuser 
>> ip.ip.ip.ip FETCH](cmd_fetch+0x31b) [0x5581ac75e1bb] -> dovecot/imap 
>> [myuser ip.ip.ip.ip FETCH](command_exec+0x5c) [0x5581ac76a09c] -> 
>> dovecot/imap [myuser ip.ip.ip.ip FETCH](+0x1b612) [0x5581ac768612] -> 
>> dovecot/imap [myuser ip.ip.ip.ip FETCH](+0x1b6ac) [0x5581ac7686ac] -> 
>> dovecot/imap [myuser ip.ip.ip.ip FETCH](client_handle_input+0x18d) 
>> [0x5581ac768a6d] -> dovecot/imap [myuser ip.ip.ip.ip 
>> FETCH](client_input+0xac) [0x5581ac768fbc] -> 
>> /usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x52) 
>> [0x7f78b3b43482] -> 
>> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x12e) 
>> [0x7f78b3b44b9e] -> 
>> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x36) 
>> [0x7f78b3b43516] -> 
>> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7f78b3b436c8] 
>> -> /usr/lib/dovecot/libdovecot.so.0(master_service_run+0x13) 
>> [0x7f78b3ac6aa3] -> dovecot/imap [myuser ip.ip.ip.ip 
>> FETCH](main+0x329) [0x5581ac75b159] -> 
>> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) 
>> [0x7f78b36bbb97] -> dovecot/imap [myuser ip.ip.ip.ip 
>> FETCH](_start+0x2a) [0x5581ac75b2fa]
>> Feb  8 08:45:37 hostname dovecot[14882]: imap(myuser): Fatal: master: 
>> service(imap): child 14135 killed with signal 6 (core dumped)
>>
>> Over all dovecot and OS versions we use same old-fashioned setup. 
>> Mails are delivered by procmail to classic mbox file.
>> This is output of doveconf -n
>>
>> # 2.2.36.1 (5d621cf65): /etc/dovecot/dovecot.conf
>> # Pigeonhole version 0.4.21 (92477967)
>> # OS: Linux 4.18.0-14-generic x86_64 Ubuntu 18.04.1 LTS
>> # Hostname: myhostname
>> auth_mechanisms = plain login
>> auth_username_chars = 
>> abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_@üÜöÖäÄ
>> auth_worker_max_count = 60
>> disable_plaintext_auth = no
>> imap_capability = +XLIST
>> lock_method = flock
>> login_greeting = mailserver ready.
>> login_log_format_elements = user=<%u> method=%m rip=%r lip=%l %c
>> mail_cache_min_mail_count = 30
>> mail_location = mbox:~/mail:INBOX=/var/mail/%u
>> mail_max_userip_connections = 30
>> mbox_read_locks = dotlock
>> mbox_very_dirty_syncs = yes
>> namespace inbox {
>>   inbox = yes
>>   location =
>>   mailbox Archiv {
>>     auto = no
>>     special_use =
>>   }
>>   mailbox Drafts {
>>     auto = no
>>     special_use =
>>   }
>>   mailbox Junk {
>>     auto = no
>>     special_use =
>>   }
>>   mailbox Sent {
>>     auto = no
>>     special_use =
>>   }
>>   mailbox Trash {
>>     auto = no
>>     special_use =
>>   }
>>   prefix =
>> }
>> passdb {
>>   args = /etc/dovecot/dovecot-sql.conf
>>   driver = sql
>> }
>> protocols = imap pop3
>> service auth-worker {
>>   idle_kill = 1 mins
>> }
>> service auth {
>>   idle_kill = 1 mins
>>   unix_listener /var/spool/postfix/private/auth {
>>     group = postfix
>>     mode = 0660
>>     user = postfix
>>   }
>>   user = root
>> }
>> service imap-login {
>>   executable = /usr/lib/dovecot/imap-login
>>   process_min_avail = 4
>>   service_count = 0
>>   user = dovecot
>> }
>> service imap {
>>   process_limit = 1024
>> }
>> service pop3-login {
>>   executable = /usr/lib/dovecot/pop3-login
>>   process_min_avail = 4
>>   service_count = 0
>>   user = dovecot
>> }
>> service pop3 {
>>   executable = /usr/lib/dovecot/pop3
>>   process_limit = 1024
>> }
>> shutdown_clients = no
>> ssl_ca = </path/cert.pem
>> ssl_cert = </path/cert.pem
>> ssl_cipher_list = 
>> ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:!DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA
>> ssl_dh_parameters_length = 2048
>> ssl_key =  # hidden, use -P to show it
>> ssl_prefer_server_ciphers = yes
>> userdb {
>>   args = /etc/dovecot/dovecot-sql.conf
>>   driver = sql
>> }
>> verbose_proctitle = yes
>> protocol imap {
>>   imap_client_workarounds = delay-newmail
>>   mail_plugins = mail_log notify
>>   ssl = yes
>> }
>> protocol pop3 {
>>   pop3_client_workarounds = oe-ns-eoh outlook-no-nuls
>>   pop3_no_flag_updates = no
>>   pop3_uidl_format = %08Xu%08Xv
>>   ssl = yes
>> }
>>
>>
>> Whats going wrong here? Main problem is first one. dovecot stops 
>> showing new mail during service without telling a reason. reason 
>> looks like a broken index file.
>> What to do here? Please tell me your opinion.
>>
>> Thanks,
>> Hajo
>>
> Thanks,
> Hajo
>
Thanks,
Hajo