Replication stalled by failed attemts to read attachments (mail_attachment_dir)

[Ralf Becker]

A separate mail-attachment store and replication seems to a very bad idea!

If anything goes wrong eg. full filesystem or an accidentality deleted 
attachment-file, you wont get your replication up again :(

Only way I found so far is greping the log for the error and use that to 
expunge the whole mails, something like:

docker logs -f dovecot 2>&1 |
grep 'Error: dsync(61c8ab10dbe7): read(attachments-connector' |
cut -d/ -f5,6,8,10 | cut -d ')' -f1 | sort | uniq | sed -e 's#/\(u.\)\?# 
#g' |
while read domain user mailbox uid; do
     doveadm expunge -u $user@$domain mailbox $mailbox uid $uid
done

(The above delete the mail from both nodes!)

While having the above running on the node with the problem, you need to 
trigger a full sync on the *other* node with:

     doveadm replicator replicate -f $user@$domain

until there are no more errors and

     watch doveadm replicator status $user@$domain

shows the mailbox in sync again.

I'm happy to hear better suggestions, specially some more automatic and 
- possibly - only removing the missing attachment and not the whole mail.

Ralf


Am 05.08.21 um 18:03 schrieb Ralf Becker:
> I'm migrating an older Dovecot 2.2.19 installation with dbox and 
> mail_attachment_dir to 2.3.15 with replication to a second server.
>
> The storage from the old server was rsync'ed to a new server running 
> Dovecot 2.3.15 in a container using similar configuration with dbox 
> and mail_attachments. That servers alone seems to run fine.
>
> As a next step I tried to establish replication to the second empty 
> server, which is configured to use mdbox and no mail_attachment_dir, 
> as the rest of my Dovecot servers. Looking at "doveadm replicator 
> status '*'" some of the mailboxes replicate correct, but roughly half 
> of them fail :(
>
> Looking a the Dovecot log with mail_debug=yes shows the replication 
> fails on the source (server with dbox and mail_attachment_dir) 
> typically with one of two errors:
>
> a) attachment has different size:
>
> Aug 05 15:41:58 doveadm: Debug: Mailbox INBOX: UID 30: Looked up field 
> date.received from mail cache
> Aug 05 15:41:58 doveadm: Error: dsync(cdaf50f7580e): 
> read(attachments-connector(/var/dovecot/imap/<domain>/<user>/mailboxes/INBOX/dbox-Mails/u.30)) 
> failed: 
> read(/var/dovecot/imap/attachments/64/4e/644ecfbc09223b8ccafc6f36aa86603cde91e76716b9b5ed0cf875bc0e183b6b4581d671f8321647be99f26d2d22088e999714529e189b25a6e7d5b428fbf5d5-1a6b701e68c9c050b10a0000838cbfe1-f86a701e68c9c050b10a0000838cbfe1-30[base64:18 
> b/l]) failed: Stream is larger than expected (262658 > 262657, eof=1) 
> (last sent=mail, last recv=mail_request (EOL))
>
> showing on the other log as:
>
> Aug 05 15:49:18 doveadm: Debug: Mailbox INBOX: UID 29: Looked up field 
> guid from mail cache
> Aug 05 15:49:19 doveadm: Error: dsync(142878d8d6bf): read() failed: 
> read(10.44.99.1) failed: dot-input stream ends without '.' line (last 
> sent=mail_request (EOL), last recv=mail)
>
> b) attachment is not found at all:
>
> Aug 05 15:43:58 doveadm(<user>@<domain> 1441): Debug: fs-sis: fs-sis: 
> fs-posix: 
> open(/var/dovecot/imap/attachments/e2/0a/e20ab165f3a2e7164520341cddb06cd053dc1a87fdd5c6360b45c20a2c9d33576c8ea2daa42062f9bb0fd94f9c5f9919ad86fb11e29effe3bdf4e853fa7c0e1d-20a0b63613e6c050620b0000838cbfe1-c8d5221c39a40b61250900007dc04144-3) 
> failed: No such file or directory
> Aug 05 15:43:58 doveadm(<user>@<domain> 1441): Error: 
> dsync(cdaf50f7580e): 
> read(attachments-connector(/var/dovecot/imap/<domain>/<user>/mailboxes/INBOX/dbox-Mails/u.3)) 
> failed: 
> read(/var/dovecot/imap/attachments/e2/0a/e20ab165f3a2e7164520341cddb06cd053dc1a87fdd5c6360b45c20a2c9d33576c8ea2daa42062f9bb0fd94f9c5f9919ad86fb11e29effe3bdf4e853fa7c0e1d-20a0b63613e6c050620b0000838cbfe1-c8d5221c39a40b61250900007dc04144-3[base64:19 
> b/l]) failed: 
> open(/var/dovecot/imap/attachments/e2/0a/e20ab165f3a2e7164520341cddb06cd053dc1a87fdd5c6360b45c20a2c9d33576c8ea2daa42062f9bb0fd94f9c5f9919ad86fb11e29effe3bdf4e853fa7c0e1d-20a0b63613e6c050620b0000838cbfe1-c8d5221c39a40b61250900007dc04144-3) 
> failed: No such file or directory (last sent=mail, last 
> recv=mail_request (EOL))
>
> That caused the replication to fail and not newer mails to be 
> replicated for the given user and mailbox, which can be verified eg. 
> with "doveadm mailbox status -u <user>@<domain> all inbox".
>
> The concerned mails are pretty old / the attachment itself can not be 
> recovered, and nothing can be done about them. But how can I repair 
> the mailbox, so all unaffected mails / attachments get replicated?
>
> I already tried multiple "doveadm force-resync -u <user>@<domain> 
> INBOX" in case a) with no visible effect / still same error.
>
> Both Dovecot configurations are attached, in case they matter ...
>
> Ralf
>

-- 
Ralf Becker
EGroupware GmbH [www.egroupware.org]
Handelsregister HRB Kaiserslautern 3587
Geschäftsführer Birgit und Ralf Becker
Leibnizstr. 17, 67663 Kaiserslautern, Germany
Telefon +49 631 31657-0