Replication stalled by failed attemts to read attachments (mail_attachment_dir)

[Ralf Becker]

Made some progress, the attachments are not lost, the new Dovecot server 
(it was a migration from 2.2.19 to 2.3.15 on a different host), searches 
them under a partially different filename:

Aug 09 08:26:19 doveadm: Error: dsync(61c8ab10dbe7): 
read(attachments-connector(/var/dovecot/imap/$domain/$user/mailboxes/INBOX/dbox-Mails/u.23306)) 
failed: 
read(/var/dovecot/imap/attachments/99/93/999382f10d91c26e28f964efd3039f42041b73c00ea71da86481476e7adb1ce5a50653b2814ba57b9e8a4f5284203f1b4d3ec0c617de04d750a42b9d0cfb7855-7ca8b3186e43f560fa190000838cbfe1-e8584c2cf5a10b61640000007dc04144-23306[base64:19 
b/l]) failed: 
open(/var/dovecot/imap/attachments/99/93/999382f10d91c26e28f964efd3039f42041b73c00ea71da86481476e7adb1ce5a50653b2814ba57b9e8a4f5284203f1b4d3ec0c617de04d750a42b9d0cfb7855-7ca8b3186e43f560fa190000838cbfe1-e8584c2cf5a10b61640000007dc04144-23306) 
failed: No such file or directory (last sent=mail, last 
recv=mail_request (EOL))

Looking at the filesystem:

root at ka-nfs-mail:~# ls -l 
/var/dovecot/imap/attachments/99/93/999382f10d91c26e28f964efd3039f42041b73c00ea71da86481476e7adb1ce5a50653b2814ba57b9e8a4f5284203f1b4d3ec0c617de04d750a42b9d0cfb7855-7ca8b3186e43f560fa190000838cbfe1-e8584c2cf5a10b61640000007dc04144-23306
ls: cannot access 
'/poolN/nordenham/attachments/99/93/999382f10d91c26e28f964efd3039f42041b73c00ea71da86481476e7adb1ce5a50653b2814ba57b9e8a4f5284203f1b4d3ec0c617de04d750a42b9d0cfb7855-7ca8b3186e43f560fa190000838cbfe1-e8584c2cf5a10b61640000007dc04144-23306': 
No such file or directory

root at ka-nfs-mail:~# ls -l 
/var/dovecot/imap/attachments/99/93/999382f10d91c26e28f964efd3039f42041b73c00ea71da86481476e7adb1ce5a50653b2814ba57b9e8a4f5284203f1b4d3ec0c617de04d750a42b9d0cfb7855-7ca8b3186e43f560fa190000838cbfe1-*-23306
-rw------- 1 90 systemd-journal 605950 Jul 19 09:18 
/var/dovecot/imap/attachments/99/93/999382f10d91c26e28f964efd3039f42041b73c00ea71da86481476e7adb1ce5a50653b2814ba57b9e8a4f5284203f1b4d3ec0c617de04d750a42b9d0cfb7855-7ca8b3186e43f560fa190000838cbfe1-578fed299419c150550c0000838cbfe1-23306

So the attachment still exists (verified by it content), but the 3rd 
dash-separated filename-part has changed:
1. configured (mail_attachment_hash) sha256 hash of the attachment-content
2. guid_128 
(https://github.com/dovecot/core/blob/a5209c83c3a82386c94d466eec5fea394973e88f/src/lib-storage/index/index-attachment.c#L114)
3. no idea, but changed
4. uid of the mail

So the questions are:
- what is that 3rd part
- how could it have changed by the migration
- is there a way to force it back to the existing file-names

Unfortunately I found no documentation but the source-code on Github 
about the inner working of that attachment store :(

Any help or ideas would be really appreciated :)

Ralf


Am 07.08.21 um 19:47 schrieb Ralf Becker:
> A separate mail-attachment store and replication seems to a very bad 
> idea!
>
> If anything goes wrong eg. full filesystem or an accidentality deleted 
> attachment-file, you wont get your replication up again :(
>
> Only way I found so far is greping the log for the error and use that 
> to expunge the whole mails, something like:
>
> docker logs -f dovecot 2>&1 |
> grep 'Error: dsync(61c8ab10dbe7): read(attachments-connector' |
> cut -d/ -f5,6,8,10 | cut -d ')' -f1 | sort | uniq | sed -e 
> 's#/\(u.\)\?# #g' |
> while read domain user mailbox uid; do
>     doveadm expunge -u $user@$domain mailbox $mailbox uid $uid
> done
>
> (The above delete the mail from both nodes!)
>
> While having the above running on the node with the problem, you need 
> to trigger a full sync on the *other* node with:
>
>     doveadm replicator replicate -f $user@$domain
>
> until there are no more errors and
>
>     watch doveadm replicator status $user@$domain
>
> shows the mailbox in sync again.
>
> I'm happy to hear better suggestions, specially some more automatic 
> and - possibly - only removing the missing attachment and not the 
> whole mail.
>
> Ralf
>
>
> Am 05.08.21 um 18:03 schrieb Ralf Becker:
>> I'm migrating an older Dovecot 2.2.19 installation with dbox and 
>> mail_attachment_dir to 2.3.15 with replication to a second server.
>>
>> The storage from the old server was rsync'ed to a new server running 
>> Dovecot 2.3.15 in a container using similar configuration with dbox 
>> and mail_attachments. That servers alone seems to run fine.
>>
>> As a next step I tried to establish replication to the second empty 
>> server, which is configured to use mdbox and no mail_attachment_dir, 
>> as the rest of my Dovecot servers. Looking at "doveadm replicator 
>> status '*'" some of the mailboxes replicate correct, but roughly half 
>> of them fail :(
>>
>> Looking a the Dovecot log with mail_debug=yes shows the replication 
>> fails on the source (server with dbox and mail_attachment_dir) 
>> typically with one of two errors:
>>
>> a) attachment has different size:
>>
>> Aug 05 15:41:58 doveadm: Debug: Mailbox INBOX: UID 30: Looked up 
>> field date.received from mail cache
>> Aug 05 15:41:58 doveadm: Error: dsync(cdaf50f7580e): 
>> read(attachments-connector(/var/dovecot/imap/<domain>/<user>/mailboxes/INBOX/dbox-Mails/u.30)) 
>> failed: 
>> read(/var/dovecot/imap/attachments/64/4e/644ecfbc09223b8ccafc6f36aa86603cde91e76716b9b5ed0cf875bc0e183b6b4581d671f8321647be99f26d2d22088e999714529e189b25a6e7d5b428fbf5d5-1a6b701e68c9c050b10a0000838cbfe1-f86a701e68c9c050b10a0000838cbfe1-30[base64:18 
>> b/l]) failed: Stream is larger than expected (262658 > 262657, eof=1) 
>> (last sent=mail, last recv=mail_request (EOL))
>>
>> showing on the other log as:
>>
>> Aug 05 15:49:18 doveadm: Debug: Mailbox INBOX: UID 29: Looked up 
>> field guid from mail cache
>> Aug 05 15:49:19 doveadm: Error: dsync(142878d8d6bf): read() failed: 
>> read(10.44.99.1) failed: dot-input stream ends without '.' line (last 
>> sent=mail_request (EOL), last recv=mail)
>>
>> b) attachment is not found at all:
>>
>> Aug 05 15:43:58 doveadm(<user>@<domain> 1441): Debug: fs-sis: fs-sis: 
>> fs-posix: 
>> open(/var/dovecot/imap/attachments/e2/0a/e20ab165f3a2e7164520341cddb06cd053dc1a87fdd5c6360b45c20a2c9d33576c8ea2daa42062f9bb0fd94f9c5f9919ad86fb11e29effe3bdf4e853fa7c0e1d-20a0b63613e6c050620b0000838cbfe1-c8d5221c39a40b61250900007dc04144-3) 
>> failed: No such file or directory
>> Aug 05 15:43:58 doveadm(<user>@<domain> 1441): Error: 
>> dsync(cdaf50f7580e): 
>> read(attachments-connector(/var/dovecot/imap/<domain>/<user>/mailboxes/INBOX/dbox-Mails/u.3)) 
>> failed: 
>> read(/var/dovecot/imap/attachments/e2/0a/e20ab165f3a2e7164520341cddb06cd053dc1a87fdd5c6360b45c20a2c9d33576c8ea2daa42062f9bb0fd94f9c5f9919ad86fb11e29effe3bdf4e853fa7c0e1d-20a0b63613e6c050620b0000838cbfe1-c8d5221c39a40b61250900007dc04144-3[base64:19 
>> b/l]) failed: 
>> open(/var/dovecot/imap/attachments/e2/0a/e20ab165f3a2e7164520341cddb06cd053dc1a87fdd5c6360b45c20a2c9d33576c8ea2daa42062f9bb0fd94f9c5f9919ad86fb11e29effe3bdf4e853fa7c0e1d-20a0b63613e6c050620b0000838cbfe1-c8d5221c39a40b61250900007dc04144-3) 
>> failed: No such file or directory (last sent=mail, last 
>> recv=mail_request (EOL))
>>
>> That caused the replication to fail and not newer mails to be 
>> replicated for the given user and mailbox, which can be verified eg. 
>> with "doveadm mailbox status -u <user>@<domain> all inbox".
>>
>> The concerned mails are pretty old / the attachment itself can not be 
>> recovered, and nothing can be done about them. But how can I repair 
>> the mailbox, so all unaffected mails / attachments get replicated?
>>
>> I already tried multiple "doveadm force-resync -u <user>@<domain> 
>> INBOX" in case a) with no visible effect / still same error.
>>
>> Both Dovecot configurations are attached, in case they matter ...
>>
>> Ralf
>>
>

-- 
Ralf Becker
EGroupware GmbH [www.egroupware.org]
Handelsregister HRB Kaiserslautern 3587
Geschäftsführer Birgit und Ralf Becker
Leibnizstr. 17, 67663 Kaiserslautern, Germany
Telefon +49 631 31657-0