Corrupted index cache file - migration with zlib
Hi
I'm migrating some mailboxes from an old dovecot system where mailboxes are stored maildir uncompressed. I enabled zlib on the new system and migrated an account: doveadm -o mail_fsync=never backup -R -u tonyg imapc:
when the account connects to the new system after a while i'm seeing the following error and the mailbox is no longer functioning:
Jan 15 13:58:06 S611 dovecot: imap(tonyg): Error: read(/srv/vmail/532868/tonyg/mdbox/storage/m.1): FETCH BODY[] for mailbox INBOX UID 61 got too little data: 4106341 vs 6054268 Jan 15 13:58:06 S611 dovecot: imap(tonyg): Error: unlink(/srv/vmail/532868/tonyg/mdbox/mailboxes/INBOX/dbox-Mails/dovecot.index.cache) failed: No such file or directory (in mail-cache.c:28) Jan 15 13:58:06 S611 dovecot: imap(tonyg): Error: Corrupted index cache file /srv/vmail/532868/tonyg/mdbox/mailboxes/INBOX/dbox-Mails/dovecot.index.cache: Broken virtual size for mail UID 61 Jan 15 13:58:06 S611 dovecot: imap(tonyg): FETCH failed in=181 out=4109653 Jan 15 13:58:06 S611 dovecot: auth-worker(6130): ldap(tonyg,144.132.1.90): unknown user Jan 15 13:58:06 S611 dovecot: imap-login: Login: user=<tonyg>, method=PLAIN, rip=144.132.1.90, lip=203.134.11.127, mpid=6746, secured, session=<7qqWklYpnGjLhgHK> Jan 15 13:58:07 S611 dovecot: imap(tonyg): Error: read(/srv/vmail/532868/tonyg/mdbox/storage/m.1): FETCH BODY[] for mailbox INBOX UID 61 got too little data: 4106341 vs 6054268 Jan 15 13:58:07 S611 dovecot: imap(tonyg): Error: unlink(/srv/vmail/532868/tonyg/mdbox/mailboxes/INBOX/dbox-Mails/dovecot.index.cache) failed: No such file or directory (in mail-cache.c:28) Jan 15 13:58:07 S611 dovecot: imap(tonyg): Error: Corrupted index cache file /srv/vmail/532868/tonyg/mdbox/mailboxes/INBOX/dbox-Mails/dovecot.index.cache: Broken virtual size for mail UID 61 Jan 15 13:58:07 S611 dovecot: imap(tonyg): FETCH failed in=181 out=4109653
If i migrate the account without zlib everthing works fine and the error can't be reproduced.
I have tried force-resync and index commands to no avail.
Any ideas what's going on here?
doveconf -n:
[root@S611 532868]# doveconf -n
# 2.2.21 (5345f22): /etc/dovecot/dovecot.conf # Pigeonhole version 0.4.11 (6ddf62b87f60+) # OS: Linux 3.10.0-327.4.4.el7.x86_64 x86_64 CentOS Linux release 7.2.1511 (Core) auth_debug = yes auth_verbose = yes imapc_features = rfc822.size fetch-headers imapc_host = removed imapc_master_user = removed imapc_password = # hidden, use -P to show it imapc_user = %u lmtp_proxy = yes login_trusted_networks = 192.168.10.0/24 mail_debug = yes mail_gid = vmail mail_location = mdbox:%h/mdbox:ALT=/altstorage%h/mdbox mail_plugins = quota zlib mail_prefetch_count = 50 mail_uid = vmail managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart mdbox_rotate_size = 10 M namespace inbox { inbox = yes location = mailbox "Deleted Messages" { special_use = \Trash } mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { args = /etc/dovecot/dovecot-ldap.conf.ext driver = ldap } passdb { args = /etc/dovecot/dovecot-ldap-m2core.conf.ext driver = ldap } plugin { quota = dict:User quota::file:%h/dovecot-quota sieve = file:~/sieve;active=~/.dovecot.sieve zlib_save = gz zlib_save_level = 6 } pop3c_host = <removed> pop3c_password = # hidden, use -P to show it service lmtp { inet_listener lmtp { address = <removed> port = 24 } } ssl_cert =
figured out my issue
I changed: protocol imap { mail_plugins = quota imap_quota imap_zlib }
to this: protocol imap { mail_plugins = $mail_plugins imap_quota imap_zlib }
No longer getting the error after migration with zlib enabled.. could there be any kind of better warning or error handling around this? (if this is the actual issue)
On Fri, Jan 15, 2016 at 2:25 PM Leon Kyneur leon@dexterous.org wrote:
Hi
I'm migrating some mailboxes from an old dovecot system where mailboxes are stored maildir uncompressed. I enabled zlib on the new system and migrated an account: doveadm -o mail_fsync=never backup -R -u tonyg imapc:
when the account connects to the new system after a while i'm seeing the following error and the mailbox is no longer functioning:
Jan 15 13:58:06 S611 dovecot: imap(tonyg): Error: read(/srv/vmail/532868/tonyg/mdbox/storage/m.1): FETCH BODY[] for mailbox INBOX UID 61 got too little data: 4106341 vs 6054268 Jan 15 13:58:06 S611 dovecot: imap(tonyg): Error: unlink(/srv/vmail/532868/tonyg/mdbox/mailboxes/INBOX/dbox-Mails/dovecot.index.cache) failed: No such file or directory (in mail-cache.c:28) Jan 15 13:58:06 S611 dovecot: imap(tonyg): Error: Corrupted index cache file /srv/vmail/532868/tonyg/mdbox/mailboxes/INBOX/dbox-Mails/dovecot.index.cache: Broken virtual size for mail UID 61 Jan 15 13:58:06 S611 dovecot: imap(tonyg): FETCH failed in=181 out=4109653 Jan 15 13:58:06 S611 dovecot: auth-worker(6130): ldap(tonyg,144.132.1.90): unknown user Jan 15 13:58:06 S611 dovecot: imap-login: Login: user=<tonyg>, method=PLAIN, rip=144.132.1.90, lip=203.134.11.127, mpid=6746, secured, session=<7qqWklYpnGjLhgHK> Jan 15 13:58:07 S611 dovecot: imap(tonyg): Error: read(/srv/vmail/532868/tonyg/mdbox/storage/m.1): FETCH BODY[] for mailbox INBOX UID 61 got too little data: 4106341 vs 6054268 Jan 15 13:58:07 S611 dovecot: imap(tonyg): Error: unlink(/srv/vmail/532868/tonyg/mdbox/mailboxes/INBOX/dbox-Mails/dovecot.index.cache) failed: No such file or directory (in mail-cache.c:28) Jan 15 13:58:07 S611 dovecot: imap(tonyg): Error: Corrupted index cache file /srv/vmail/532868/tonyg/mdbox/mailboxes/INBOX/dbox-Mails/dovecot.index.cache: Broken virtual size for mail UID 61 Jan 15 13:58:07 S611 dovecot: imap(tonyg): FETCH failed in=181 out=4109653
If i migrate the account without zlib everthing works fine and the error can't be reproduced.
I have tried force-resync and index commands to no avail.
Any ideas what's going on here?
doveconf -n:
[root@S611 532868]# doveconf -n
# 2.2.21 (5345f22): /etc/dovecot/dovecot.conf # Pigeonhole version 0.4.11 (6ddf62b87f60+) # OS: Linux 3.10.0-327.4.4.el7.x86_64 x86_64 CentOS Linux release 7.2.1511 (Core) auth_debug = yes auth_verbose = yes imapc_features = rfc822.size fetch-headers imapc_host = removed imapc_master_user = removed imapc_password = # hidden, use -P to show it imapc_user = %u lmtp_proxy = yes login_trusted_networks = 192.168.10.0/24 mail_debug = yes mail_gid = vmail mail_location = mdbox:%h/mdbox:ALT=/altstorage%h/mdbox mail_plugins = quota zlib mail_prefetch_count = 50 mail_uid = vmail managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart mdbox_rotate_size = 10 M namespace inbox { inbox = yes location = mailbox "Deleted Messages" { special_use = \Trash } mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { args = /etc/dovecot/dovecot-ldap.conf.ext driver = ldap } passdb { args = /etc/dovecot/dovecot-ldap-m2core.conf.ext driver = ldap } plugin { quota = dict:User quota::file:%h/dovecot-quota sieve = file:~/sieve;active=~/.dovecot.sieve zlib_save = gz zlib_save_level = 6 } pop3c_host = <removed> pop3c_password = # hidden, use -P to show it service lmtp { inet_listener lmtp { address = <removed> port = 24 } } ssl_cert =
On 1/14/16 8:51 PM, Leon Kyneur wrote:
figured out my issue
I changed: protocol imap { mail_plugins = quota imap_quota imap_zlib }
to this: protocol imap { mail_plugins = $mail_plugins imap_quota imap_zlib }
Right. This is because it needs the "zlib" plugin enabled, and in your original configuration, the zlib plugin was enabled systemwide:
mail_plugins = quota zlib
... but these lines completely replaced the "mail_plugins" setting for imap, and didn't include zlib:
protocol imap { mail_plugins = quota imap_quota imap_zlib }
Your fixed configuration *adds* additional plugins to the systemwide mail_plugins setting rather than replacing them, so it correctly uses zlib for IMAP. It would also work if you used:
protocol imap { mail_plugins = quota zlib imap_quota imap_zlib }
No longer getting the error after migration with zlib enabled.. could there be any kind of better warning or error handling around this? (if this is the actual issue)
Well, the documentation at http://wiki.dovecot.org/Plugins/Zlib does suggest doing it with "mail_plugins = $mail_plugins zlib", and it gives those errors in the log if you forget it.
There's an argument to be made for "Dovecot should *always* try to uncompress 'obviously' compressed files", that apparently opens up security issues, exposing Dovecot to bugs in the system's compression libraries even if the admin intentionally doesn't use compression.
-- Robert L Mathews, Tiger Technologies, http://www.tigertech.net/
participants (2)
-
Leon Kyneur
-
Robert L Mathews