<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">On 8. Sep 2020, at 12.35, Robert Nowotny <<a href="mailto:rnowotny1966@gmail.com" class="">rnowotny1966@gmail.com</a>> wrote:<br class=""><div><blockquote type="cite" class=""><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><div class="gmail-moz-text-html" lang="x-unicode">
<br class="">
<div class="gmail-moz-forward-container">
<font face="Courier New" class="">Dear Aki,<br class="">
I switched to "gz" now, since "zstd" also gave some errors on
writing to files.<br class=""></font></div></div></div></div></blockquote><div><br class=""></div><div>What kind of errors?</div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class="gmail-moz-text-html" lang="x-unicode"><div class="gmail-moz-forward-container"><font face="Courier New" class="">
I dont know if "xz" compression or "zstd" shreddered my MDBOX
Files, but I lost 4 days of mail. (a couple of thousand mails).<br class="">
After restoring the backup (what was made after switching to
version 2.3.11.3) I still have some broken mdfiles, but not too
many.<br class="">
Interestingly always in the "Sent" Mailbox on a number of Users.<br class="">
<br class="">
I just can not go back to 20.08.2020 before I updated to </font><font face="Courier New" class=""><font face="Courier New" class="">2.3.11.3</font> -
too many emails would have been lost.<br class="">
<br class="">
So - the current status is </font><font face="Courier New" class=""><font face="Courier New" class=""><font face="Courier New" class="">2.3.11.3</font></font>,
with "gz" compression.<br class="">
<br class="">
</font><font face="Courier New" class="">I force-synced and re-indexed all
the mdbox files (250 GB), but still have some broken - please
how can I fix those ?<br class=""></font></div></div></div></div></blockquote><div><br class=""></div><div>Note that force-resync doesn't read through the mails to verify whether they are readable. It just verifies that the indexes and metadata is ok. The only way to verify that all mails are readable is to actually try to read all of their text (e.g. doveadm fetch -u user text mailbox Sent > /dev/null).</div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class="gmail-moz-text-html" lang="x-unicode"><div class="gmail-moz-forward-container"><font face="Courier New" class="">
You stated : <br class="">
</font><br class="">
<pre style="font-family: courier, "courier new", monospace; font-size: 14px; white-space: pre-wrap; margin-top: 0px; margin-bottom: 0px;" class="">(In theory you could leave the existing mails xz-compressed, but best
would be to re-compress everything via dsync so old mails can be read
when we eventually remove xz support.)
</pre>
<br class="gmail-Apple-interchange-newline">
what is the optimal way to do that, expecially not loosing the
index (?) for outlook/thunderbird, for third party tools that rely
on some index/hash (dunno how exactly that works) ? <br class=""></div></div></div></div></blockquote><div><br class=""></div><div>IMAP clients use the folder's UIDVALIDITY and message UID numbers to preserve caches. Using dsync preserves these. See doveadm-sync man page. It's also in <a href="https://wiki.dovecot.org/Tools/Doveadm/Sync" class="">https://wiki.dovecot.org/Tools/Doveadm/Sync</a></div><div><br class=""></div><div>From your previous mail:</div><div><br class=""></div><div><blockquote type="cite" class="">sudo doveadm backup -D -u "${mailbox_username}" "mdbox:/home/vmail/virtualmailboxes/${mailbox_username}_backup"<br class="">sudo service dovecot stop<br class="">sudo mv "/home/vmail/virtualmailboxes/${mailbox_username}" "/home/vmail/virtualmailboxes/${mailbox_username}_original" <br class="">sudo mv "/home/vmail/virtualmailboxes/${mailbox_username}_backup" "/home/vmail/virtualmailboxes/${mailbox_username}" <br class="">sudo service dovecot start<br class=""></blockquote><br class=""></div><div>The problem is that in your config you have:</div><div><br class=""></div><div>mail_location = mdbox:/home/vmail/virtualmailboxes/%n:DIRNAME=dbox-Mails-nocollision-random-KOKxNmMJkEBeCitBhFwS<br class=""><br class=""></div><div>You need to preserve the DIRNAME. So:</div><div><br class=""></div><div>sudo doveadm backup -D -u "${mailbox_username}" "mdbox:/home/vmail/virtualmailboxes/${mailbox_username}_backup:DIRNAME=dbox-Mails-nocollision-random-KOKxNmMJkEBeCitBhFwS"</div><div>sudo service dovecot stop<br class="">sudo mv "/home/vmail/virtualmailboxes/${mailbox_username}" "/home/vmail/virtualmailboxes/${mailbox_username}_original" <br class="">sudo mv "/home/vmail/virtualmailboxes/${mailbox_username}_backup" "/home/vmail/virtualmailboxes/${mailbox_username}" <br class="">sudo service dovecot start<br class=""><br class=""></div><div>However, the dsync will likely fail as well because it can't read some of the mails. So you'll need to fix those in any case first. That's a bit trickier:</div><div><br class=""></div><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class="gmail-moz-text-html" lang="x-unicode"><div class="gmail-moz-forward-container"><font face="Courier New" class="">Sep 3 08:33:25 lxc-imap dovecot:
imap(mpaul)<48684><2/9E5mKuAezAqKjk>: Error: Mailbox
Sent: UID=2171:
read(zlib(/home/vmail/virtualmailboxes/mpaul/storage/m.119))
failed: read(/home/vmail/virtualmailboxes/mpaul/storage/m.119)
failed: Broken pipe (FETCH BODY[])<br class=""></font></div></div></div></div></blockquote><div><br class=""></div><div>For example here the best fix would be to try to preserve the mail as best as possible:</div><div><br class=""></div><div>a) Preserve as much of the text as possible and expunge the broken mail:</div><div><br class=""></div><div>doveadm fetch -u mpaul text mailbox Sent uid 2171 > msg.broken</div>doveadm expunge -u mpaul mailbox Sent uid 2171</div><div>doveadm save -u mpaul -m mailbox < msg.broken</div><div><br class=""></div><div>b) You could also see if the issue is that Dovecot just can't read a properly compressed email, or if the issue was that it wrote broken emails. You can extract the raw compressed mail with:</div><div><br class=""></div><div>doveadm -o mail_plugins=virtual fetch -u mpaul text mailbox Sent uid 2171 | tail -n +2 > msg.broken</div><div>cat msg.broken | xz -d</div><div><br class=""></div><div>Also this way you can see if the broken mail is actually xz or zstd or zlib. It would be nice to know if there are any zstd or zlib compressed mails that have problems. We did a lot of stress testing with zstd and also with xz, but haven't been able to reproduce any problems. It's also strange that it says taht the error is "Broken pipe" - that doesn't indicate that the mail is corrupted but that there is something more strange going on. So perhaps you don't actually have any mails written as corrupted, but Dovecot is just somehow having trouble reading the mails.</div><div><br class=""></div></body></html>