zlib errors after upgrading to 2.3.11.3

[Timo Sirainen]

On 8. Sep 2020, at 12.35, Robert Nowotny <rnowotny1966 at gmail.com> wrote:
> 
> 
> Dear Aki,
> I switched to "gz" now, since "zstd" also gave some errors on writing to files.

What kind of errors?

> I dont know if "xz" compression or "zstd" shreddered my MDBOX Files, but I lost 4 days of mail. (a couple of thousand mails).
> After restoring the backup (what was made after switching to version 2.3.11.3) I still have some broken mdfiles, but not too many.
> Interestingly always in the "Sent" Mailbox on a number of Users.
> 
> I just can not go back to 20.08.2020 before I updated to 2.3.11.3 - too many emails would have been lost.
> 
> So - the current status is 2.3.11.3, with "gz" compression.
> 
> I force-synced and re-indexed all the mdbox files (250 GB), but still have some broken - please how can I fix those ?

Note that force-resync doesn't read through the mails to verify whether they are readable. It just verifies that the indexes and metadata is ok. The only way to verify that all mails are readable is to actually try to read all of their text (e.g. doveadm fetch -u user text mailbox Sent > /dev/null).

> You stated : 
> 
> (In theory you could leave the existing mails xz-compressed, but best 
> would be to re-compress everything via dsync so old mails can be read 
> when we eventually remove xz support.)
> 
> what is the optimal way to do that, expecially not loosing the index (?) for outlook/thunderbird, for third party tools that rely on some index/hash (dunno how exactly that works) ? 

IMAP clients use the folder's UIDVALIDITY and message UID numbers to preserve caches. Using dsync preserves these. See doveadm-sync man page. It's also in https://wiki.dovecot.org/Tools/Doveadm/Sync <https://wiki.dovecot.org/Tools/Doveadm/Sync>

From your previous mail:

> sudo doveadm backup -D -u "${mailbox_username}" "mdbox:/home/vmail/virtualmailboxes/${mailbox_username}_backup"
> sudo service dovecot stop
> sudo mv "/home/vmail/virtualmailboxes/${mailbox_username}" "/home/vmail/virtualmailboxes/${mailbox_username}_original" 
> sudo mv "/home/vmail/virtualmailboxes/${mailbox_username}_backup" "/home/vmail/virtualmailboxes/${mailbox_username}" 
> sudo service dovecot start

The problem is that in your config you have:

mail_location = mdbox:/home/vmail/virtualmailboxes/%n:DIRNAME=dbox-Mails-nocollision-random-KOKxNmMJkEBeCitBhFwS

You need to preserve the DIRNAME. So:

sudo doveadm backup -D -u "${mailbox_username}" "mdbox:/home/vmail/virtualmailboxes/${mailbox_username}_backup:DIRNAME=dbox-Mails-nocollision-random-KOKxNmMJkEBeCitBhFwS"
sudo service dovecot stop
sudo mv "/home/vmail/virtualmailboxes/${mailbox_username}" "/home/vmail/virtualmailboxes/${mailbox_username}_original" 
sudo mv "/home/vmail/virtualmailboxes/${mailbox_username}_backup" "/home/vmail/virtualmailboxes/${mailbox_username}" 
sudo service dovecot start

However, the dsync will likely fail as well because it can't read some of the mails. So you'll need to fix those in any case first. That's a bit trickier:

> Sep  3 08:33:25 lxc-imap dovecot: imap(mpaul)<48684><2/9E5mKuAezAqKjk>: Error: Mailbox Sent: UID=2171: read(zlib(/home/vmail/virtualmailboxes/mpaul/storage/m.119)) failed: read(/home/vmail/virtualmailboxes/mpaul/storage/m.119) failed: Broken pipe (FETCH BODY[])

For example here the best fix would be to try to preserve the mail as best as possible:

a) Preserve as much of the text as possible and expunge the broken mail:

doveadm fetch -u mpaul text mailbox Sent uid 2171 > msg.broken
doveadm expunge -u mpaul mailbox Sent uid 2171
doveadm save -u mpaul -m mailbox < msg.broken

b) You could also see if the issue is that Dovecot just can't read a properly compressed email, or if the issue was that it wrote broken emails. You can extract the raw compressed mail with:

doveadm -o mail_plugins=virtual fetch -u mpaul text mailbox Sent uid 2171 | tail -n +2 > msg.broken
cat msg.broken | xz -d

Also this way you can see if the broken mail is actually xz or zstd or zlib. It would be nice to know if there are any zstd or zlib compressed mails that have problems. We did a lot of stress testing with zstd and also with xz, but haven't been able to reproduce any problems. It's also strange that it says taht the error is "Broken pipe" - that doesn't indicate that the mail is corrupted but that there is something more strange going on. So perhaps you don't actually have any mails written as corrupted, but Dovecot is just somehow having trouble reading the mails.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://dovecot.org/pipermail/dovecot/attachments/20200908/6e28b459/attachment.html>