Using attachment_dir with plugin zlib corrupt mails

Tue Oct 22 21:10:12 EEST 2019

at least for me it didn't fix. still see in logs:
	Error: lzma.read(/var/spool/mail/telsch/storage/m.11800): corrupted
data at 110015

before i upgrade from 2.3.4.1 to 2.3.8 + your patch i noticed i can
access the mails with alpine and roundcube but not with thunderbird.

On 10.10.19 11:38, Timo Sirainen via dovecot wrote:
> Can you test if
> https://github.com/dovecot/core/commit/5068b11e594ad7cc1f7cedf2bd9280520e0e534d.patch fixes
> it for you?
>
>> On 10 Oct 2019, at 11.34, MAREN ZUBIZARRETA via dovecot
>> <dovecot at dovecot.org <mailto:dovecot at dovecot.org>> wrote:
>>
>> Hello:
>>   I have found the same problem reported above by Patrick Cernko
>> affecting our system and corrupting our messages. Even worse, Outlook
>> 2016 will no synchronize and the clients cannot see any message, even
>> if there is only one corrupted mail per mailbox.
>>   I cannot figure out a feasible workaround for our system, and I can
>> see that in new version 2.38 the bug is not fixed.
>> Will this issue be treated soon?
>> Thanks a lot
>>      Maren Zubizarreta
>>
>>
>>   WARNING: using attachment_dir with plugin zlib can corrupt mails
>>
>> *Patrick Cernko*pcernko at mpi-klsb.mpg.de
>> <mailto:dovecot%40dovecot.org?Subject=Re:%20Re%3A%20WARNING%3A%20using%20attachment_dir%20with%20plugin%20zlib%20can%20corrupt%20mails&In-Reply-To=%3C1ab8edcc-bef4-3b41-d09a-c523674a5031%40mpi-klsb.mpg.de%3E>
>> /Fri Jul 19 17:52:37 EEST 2019/
>>
>>   * Previous message: index worker 2.3.7 undefined symbol errors
>>     <https://dovecot.org/list/dovecot/2019-July/116617.html>
>>   * Next message: Address family not supported by protocol
>>     <https://dovecot.org/list/dovecot/2019-July/116571.html>
>>   * *Messages sorted by:*[ date ]
>>     <https://dovecot.org/list/dovecot/2019-July/date.html#116570>[
>>     thread ]
>>     <https://dovecot.org/list/dovecot/2019-July/thread.html#116570>[
>>     subject ]
>>     <https://dovecot.org/list/dovecot/2019-July/subject.html#116570>[
>>     author ]
>>     <https://dovecot.org/list/dovecot/2019-July/author.html#116570>
>>
>> ------------------------------------------------------------------------
>> Hello list, hello Dovecot developers,
>> this week, I discovered a serious bug in Dovecot, that lead to several
>> broken mails on our servers. The bug corrupts the first few characters
>> of the mail header during saving. On our setup, it was almost always
>> only the very first line of text, that was corrupted.
>> Depending on the IMAP client (they seem to request different header
>> fields, ... during mail access), the bug causes the imap process to hang
>> up the TCP connection and log errors like this:
>> >/imap(USERNAME)<4767><TeQP4ASOTK5/AAAB>: Error: Corrupted record in
>> index cache file
>> /IMAP/mail/mailboxes/USERNAME/mdbox/mailboxes/Trash/dbox-Mails/dovecot.index.cache:
>> UID 489113: Broken fields in mailbox Trash:
>> read(attachments-connector(zlib(/IMAP/mail/mailboxes/USERNAME/mdbox/storage/m.813))):
>> FETCH BODY[HEADER.FIELDS (RETURN-PATH SUBJECT)] got too little data: 2
>> vs 122/
>> In our case that finally grabbed my attention, the client was the users
>> iphone that did not display any new messages but his Thunderbird did.
>> The bug seems to be triggered by a bad "interaction" of attachment_dir
>> option and zlib plugin. If you use both, you most likely are affected,
>> too, except you only use zlib plugin for reading previously compressed
>> stored mails. That's also the workaround we use now: zlib plugin only
>> enabled in mail_plugins but no plugin/zlib_save set.
>> The bug occurs on very specific mails. Due to privacy reasons I could
>> not provide sample mails here. Storing such mails seems to trigger the
>> bug reproducible.
>> I attached a very minimal doveconf -n config, that can be used to
>> trigger the bug. If one of the developers is interested, I can try to
>> generate an "anonymized" version of such a specific mail that still
>> causes the issue. I discovered the bug on our productive systems,
>> running latest Dovecot 2.2 release, but the latest 2.3 I used during
>> debugging is affected, too.
>> During debugging, I also found one hint, that might help find the bug:
>> If you store a problematic mail with zlib_save=gz (or zlib_save=bz2) and
>> then disable the zlib plugin in mail_plugins, you can call
>> doveadm fetch -u test hdr all | grep -v ^hdr: | gzip --decompress
>> on test's mailbox with only that one broken mail.
>> This will display the beginning of the rfc822 mail text until gzip
>> terminates with "gzip: stdin: unexpected end of file", approximately
>> after twice the length of the mail HEADER. This might indicate, that
>> dovecot stores the uncompressed size of the header in it's data
>> structures although the mail is stored compressed.
>> I also found a very efficient way to find all affected mails in our setup:
>> doveadm -f flow fetch -A 'user guid mailbox uid seq flags hdr' all | \
>>    grep -a "^[^ ]+ user=" | \
>>    grep -avF ' hdr=Return-path: ' | \
>>    grep -av '.* hdr=[[:print:][:space:]]*$'
>> (runtime for ~6M mails on our servers was 20-30min)
>> This can be even more optimized if you have a powerful storage system
>> with GNU parallel:
>> >/doveadm user '*' | parallel "doveadm -f flow fetch -u '{}' 'user guid
>> mailbox uid seq flags hdr' all | grep -a '^user=' | grep -avF '
>> hdr=Return-path: ' | grep -av '.* hdr=[[:print:][:space:]]*$' || true"/
>> (runtime for ~6M mails on our servers was ~4min)
>> The command will give you a list of mails that possibly are affected,
>> check the full output of
>> doveadm fetch -u USERNAME hdr guid GUID | less
>> to verify that the header is really broken.
>> On our systems I found 39 mails within ~12M mails.
>> I was able to recover these mails "manually" by reconstructing the
>> Return-Path header line, importing the fixed mails and expunging the
>> corrupt ones. Before importing, I had to disable zlib_save option
>> obviously.
>> Best regards,
>> --
>> Patrick Cernko <pcernko at mpi-klsb.mpg.de
>> <https://dovecot.org/mailman/listinfo/dovecot>> +49 681 9325 5815
>> Joint Administration: Information Services and Technology
>> Max-Planck-Institute fuer Informatik & Softwaresysteme
>> -------------- next part --------------
>> # 2.3.6.1 (d124cc84b): /etc/dovecot/dovecot.conf
>> # OS: Linux 4.14.127.1.amd64-smp x86_64 Debian 9.9
>> # Hostname: adove.mpi-klsb.mpg.de <http://adove.mpi-klsb.mpg.de/>
>> listen = *
>> mail_attachment_dir = /var/vmail/attachments
>> mail_attachment_fs = posix
>> mail_gid = nogroup
>> mail_home = /var/vmail/%u
>> mail_location = mdbox:~/mdbox
>> mail_plugins = " zlib"
>> mail_uid = nobody
>> passdb {
>>   args = /etc/dovecot/userdb
>>   driver = passwd-file
>> }
>> plugin {
>>   zlib_save = gz
>> }
>> protocols = imap
>> userdb {
>>   args = /etc/dovecot/userdb
>>   driver = passwd-file
>> }
>> -------------- next part --------------
>> A non-text attachment was scrubbed...
>> Name: smime.p7s
>> Type: application/pkcs7-signature
>> Size: 5324 bytes
>> Desc: S/MIME Cryptographic Signature
>> URL:
>> <https://dovecot.org/pipermail/dovecot/attachments/20190719/3da620ad/attachment-0001.p7s>
>