Using attachment_dir with plugin zlib corrupt mails

Thu Oct 10 11:34:54 EEST 2019

Hello:

  I have found the same problem reported above by Patrick Cernko affecting our system and corrupting our messages. Even worse, Outlook 2016 will no synchronize and the clients cannot see any message, even if there is only one corrupted mail per mailbox.

  I cannot figure out a feasible workaround for our system, and I can see that in new version 2.38 the bug is not fixed.

 Will this issue be treated soon?

 Thanks a lot

     Maren Zubizarreta

WARNING: using attachment_dir with plugin zlib can corrupt mails
Patrick Cernko pcernko at mpi-klsb.mpg.de<mailto:dovecot%40dovecot.org?Subject=Re:%20Re%3A%20WARNING%3A%20using%20attachment_dir%20with%20plugin%20zlib%20can%20corrupt%20mails&In-Reply-To=%3C1ab8edcc-bef4-3b41-d09a-c523674a5031%40mpi-klsb.mpg.de%3E>
Fri Jul 19 17:52:37 EEST 2019

  *   Previous message: index worker 2.3.7 undefined symbol errors<https://dovecot.org/list/dovecot/2019-July/116617.html>
  *   Next message: Address family not supported by protocol<https://dovecot.org/list/dovecot/2019-July/116571.html>
  *   Messages sorted by: [ date ]<https://dovecot.org/list/dovecot/2019-July/date.html#116570> [ thread ]<https://dovecot.org/list/dovecot/2019-July/thread.html#116570> [ subject ]<https://dovecot.org/list/dovecot/2019-July/subject.html#116570> [ author ]<https://dovecot.org/list/dovecot/2019-July/author.html#116570>

________________________________

Hello list, hello Dovecot developers,

this week, I discovered a serious bug in Dovecot, that lead to several

broken mails on our servers. The bug corrupts the first few characters

of the mail header during saving. On our setup, it was almost always

only the very first line of text, that was corrupted.

Depending on the IMAP client (they seem to request different header

fields, ... during mail access), the bug causes the imap process to hang

up the TCP connection and log errors like this:

> imap(USERNAME)<4767><TeQP4ASOTK5/AAAB>: Error: Corrupted record in index cache file /IMAP/mail/mailboxes/USERNAME/mdbox/mailboxes/Trash/dbox-Mails/dovecot.index.cache: UID 489113: Broken fields in mailbox Trash: read(attachments-connector(zlib(/IMAP/mail/mailboxes/USERNAME/mdbox/storage/m.813))): FETCH BODY[HEADER.FIELDS (RETURN-PATH SUBJECT)] got too little data: 2 vs 122

In our case that finally grabbed my attention, the client was the users

iphone that did not display any new messages but his Thunderbird did.

The bug seems to be triggered by a bad "interaction" of attachment_dir

option and zlib plugin. If you use both, you most likely are affected,

too, except you only use zlib plugin for reading previously compressed

stored mails. That's also the workaround we use now: zlib plugin only

enabled in mail_plugins but no plugin/zlib_save set.

The bug occurs on very specific mails. Due to privacy reasons I could

not provide sample mails here. Storing such mails seems to trigger the

bug reproducible.

I attached a very minimal doveconf -n config, that can be used to

trigger the bug. If one of the developers is interested, I can try to

generate an "anonymized" version of such a specific mail that still

causes the issue. I discovered the bug on our productive systems,

running latest Dovecot 2.2 release, but the latest 2.3 I used during

debugging is affected, too.

During debugging, I also found one hint, that might help find the bug:

If you store a problematic mail with zlib_save=gz (or zlib_save=bz2) and

then disable the zlib plugin in mail_plugins, you can call

doveadm fetch -u test hdr all | grep -v ^hdr: | gzip --decompress

on test's mailbox with only that one broken mail.

This will display the beginning of the rfc822 mail text until gzip

terminates with "gzip: stdin: unexpected end of file", approximately

after twice the length of the mail HEADER. This might indicate, that

dovecot stores the uncompressed size of the header in it's data

structures although the mail is stored compressed.

I also found a very efficient way to find all affected mails in our setup:

doveadm -f flow fetch -A 'user guid mailbox uid seq flags hdr' all | \

   grep -a "^[^ ]+ user=" | \

   grep -avF ' hdr=Return-path: ' | \

   grep -av '.* hdr=[[:print:][:space:]]*$'

(runtime for ~6M mails on our servers was 20-30min)

This can be even more optimized if you have a powerful storage system

with GNU parallel:

> doveadm user '*' | parallel "doveadm -f flow fetch -u '{}' 'user guid mailbox uid seq flags hdr' all | grep -a '^user=' | grep -avF ' hdr=Return-path: ' | grep -av '.* hdr=[[:print:][:space:]]*$' || true"

(runtime for ~6M mails on our servers was ~4min)

The command will give you a list of mails that possibly are affected,

check the full output of

doveadm fetch -u USERNAME hdr guid GUID | less

to verify that the header is really broken.

On our systems I found 39 mails within ~12M mails.

I was able to recover these mails "manually" by reconstructing the

Return-Path header line, importing the fixed mails and expunging the

corrupt ones. Before importing, I had to disable zlib_save option obviously.

Best regards,

--

Patrick Cernko <pcernko at mpi-klsb.mpg.de<https://dovecot.org/mailman/listinfo/dovecot>> +49 681 9325 5815

Joint Administration: Information Services and Technology

Max-Planck-Institute fuer Informatik & Softwaresysteme

-------------- next part --------------

# 2.3.6.1 (d124cc84b): /etc/dovecot/dovecot.conf

# OS: Linux 4.14.127.1.amd64-smp x86_64 Debian 9.9

# Hostname: adove.mpi-klsb.mpg.de

listen = *

mail_attachment_dir = /var/vmail/attachments

mail_attachment_fs = posix

mail_gid = nogroup

mail_home = /var/vmail/%u

mail_location = mdbox:~/mdbox

mail_plugins = " zlib"

mail_uid = nobody

passdb {

  args = /etc/dovecot/userdb

  driver = passwd-file

}

plugin {

  zlib_save = gz

}

protocols = imap

userdb {

  args = /etc/dovecot/userdb

  driver = passwd-file

}

-------------- next part --------------

A non-text attachment was scrubbed...

Name: smime.p7s

Type: application/pkcs7-signature

Size: 5324 bytes

Desc: S/MIME Cryptographic Signature

URL: <https://dovecot.org/pipermail/dovecot/attachments/20190719/3da620ad/attachment-0001.p7s>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://dovecot.org/pipermail/dovecot/attachments/20191010/508eeb57/attachment-0001.html>