Broken uidlist when using NFS on newer kernels

[Jeremy Hanmer]

I know this has been reported in the past, but I think I have some useful
new information on the problem. After an OS upgrade from Ubuntu Xenial
(4.4.0 kernel) to Ubuntu Focal (5.4.0 kernel) and corresponding upgrade
from Dovecot 2.2.27 to 2.3.7.2, we've started seeing broken uidlist files
to an extent that's making larger mail boxes nearly unusable because the
file is constantly being regenerated. I've also used the
2.3.16-2+ubuntu20.04 version distributed from dovecot.org and the behavior
is unchanged. The environment consists of NFS mounts from a NetApp device,
with a couple dozen MX servers receiving mail and about a hundred IMAP/POP
servers.

This is the exact error (note the blank after "invalid data"):
Error: Mailbox INBOX: Broken file
/mnt/morty/morty2/gravest/x15775549/Maildir/dovecot-uidlist line 373:
Invalid data:

I've been able to trigger the problem rather easily by piping an email to
dovecot-lda in a loop and reading the resulting dovecot-uidlist file on a
different server. What it shows is that occasionally we're seeing the last
line of the file prepended with a number of null bytes equal to the line
that's being written (for example, if the entry is "35322
:1633719038.M516419P3623238.pdx1-sub0-mail-mx202,S=2777,W=2832", we'll have
it prepended by 69 null bytes). This then breaks the IMAP process' ability
to read the file. My first thought was to extend the retry functionality so
the imap proces makes more attempts to read the file when it detects a
problem like this, but would love input from someone more familiar with the
codebase.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://dovecot.org/pipermail/dovecot/attachments/20211008/b93b561b/attachment.html>