To whom it may interest;

With the help of Aki Tuomi I've found a way to remove such errors and move forward, in a way that could be automated. As this might be a problem to others and there seems to be no discussion about it, i'll share it with you.

What I did, essentially, was to write a shell script that do the following, per user:

• read all the mailboxes with doveadm fetch -u $username text all and redirect errors to a file
• identify all missing attachments' paths from the file created previously and try to create a hardlink to it. Any file with the same hash (before -) is good.
• identify all mailboxes and uids from messages there are still broken (the same error file created before should have this information) and fetch them, and save them elsewhere.
• after fetching and saving, expunge such messages.
• use doveadm save to put the messages back. They'll be without the attachments but also without any errors.

There are some gotchas to do the above, and to automate that, so I'll be happy to help if anyone needs.

Thank you.

On Wed, Oct 5, 2016 at 3:59 PM Webert de Souza Lima <webert.boss@gmail.com> wrote:

Hi, I've sent some e-mails about this before but since there was no answers I'll write it differently, with different information.

I'm using SIS (Single Instance Attachment Storage). For any reason that is not relevant now, many attachments are missing and the messages can't be fetched:

Error: read(attachments-connector(zlib(/dovecot/mdbox/bar.example/foo/storage/m.1))) failed: read(/dovecot/attach/bar.example/23/ae/23aed008c1f32f048afd38d9aae68c5aeae2d17a9170e28c60c75a02ec199ef4e7079cd92988ad857bd6e12cd24cdd7619bd29f26edeec842a6911bb14a86944-fb0b6a214dfa63573c1f00009331bd36[base64:19 b/l]) failed: open(/dovecot/attach/bar.example/23/ae/23aed008c1f32f048afd38d9aae68c5aeae2d17a9170e28c60c75a02ec199ef4e7079cd92988ad857bd6e12cd24cdd7619bd29f26edeec842a6911bb14a86944-fb0b6a214dfa63573c1f00009331bd36) failed: No such file or directory

in this specific case, the /dovecot/attach/bar.example/23/ae/ director doesn't exist. In other cases, just one file is missing so I would assume the hardlink could be recreated and it would work.

If I create the missing file (with touch or whatever), I get the following errors: Error: read(/dovecot/attach/bar.example/23/ae/23aed008c1f32f048afd38d9aae68c5aeae2d17a9170e28c60c75a02ec199ef4e7079cd92988ad857bd6e12cd24cdd7619bd29f26edeec842a6911bb14a86944-fb0b6a214dfa63573c1f00009331bd36[base64:19 b/l]) failed: Stream is smaller than expected (0 < 483065) Error: read(attachments-connector(zlib(/dovecot/mdbox/bar.example/foo/storage/m.1))) failed: read(/dovecot/attach/bar.example/23/ae/23aed008c1f32f048afd38d9aae68c5aeae2d17a9170e28c60c75a02ec199ef4e7079cd92988ad857bd6e12cd24cdd7619bd29f26edeec842a6911bb14a86944-fb0b6a214dfa63573c1f00009331bd36[base64:19 b/l]) failed: Stream is smaller than expected (0 < 483065) Error: fetch(body) failed for box=INBOX uid=15: BUG: Unknown internal error

If I try to fill the file with the amount of bytes it complains about with the following command:

$ dd if=/dev/zero of=/dovecot/attach/bar.example/23/ae/23aed008c1f32f048afd38d9aae68c5aeae2d17a9170e28c60c75a02ec199ef4e7079cd92988ad857bd6e12cd24cdd7619bd29f26edeec842a6911bb14a86944-fb0b6a214dfa63573c1f00009331bd36 bs=1 count=483065

then I get the following error:

Error: read(/dovecot/attach/bar.example/23/ae/23aed008c1f32f048afd38d9aae68c5aeae2d17a9170e28c60c75a02ec199ef4e7079cd92988ad857bd6e12cd24cdd7619bd29f26edeec842a6911bb14a86944-fb0b6a214dfa63573c1f00009331bd36[base64:19 b/l]) failed: Stream is larger than expected (483928 > 483065, eof=0) Error: read(attachments-connector(zlib(/srv/dovecot/mdbox/bar.example/foo/storage/m.1))) failed: read(//dovecot/attach/bar.example/23/ae/23aed008c1f32f048afd38d9aae68c5aeae2d17a9170e28c60c75a02ec199ef4e7079cd92988ad857bd6e12cd24cdd7619bd29f26edeec842a6911bb14a86944-fb0b6a214dfa63573c1f00009331bd36[base64:19 b/l]) failed: Stream is larger than expected (483928 > 483065, eof=0) Error: fetch(body) failed for box=INBOX uid=15: BUG: Unknown internal error

Based on this I have a few questions: that have compromised attachments?

1. Is there a way, or a tool to scan all mailboxes to get all the messages

2. is there a way to "fix" the missing files (even if creating fake files or removing the attachments information from the messages)

3. What I need is to migrate these boxes using doveadm backup/sync, which fails when these errors occur. Is is possible to ignore them or is there another tool that would do it?

Thank you.

Webert Lima Belo Horizonte, Brasil