<div dir="ltr"><div dir="auto">I looked into LMTP, but reconfiguring our 1.5 million mailboxes just to work around what seems like an obvious bug in the code is a hard sell. I patched maildir-uidlist.c to strip out the leading null bytes and things seem to behave just fine, but it feels wrong and I was hoping to get input from someone more familiar with the codebase.</div></div><div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Oct 12, 2021 at 8:39 AM Alessio Cecchi <<a href="mailto:alessio@skye.it" target="_blank">alessio@skye.it</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>Hi Jeremy,</p>
<p>I had the same problem as you.</p>
<p>We run an email hosting service with Maildir on NetApp NFS,
Dovecot Director and Backend servers for POP/IMAP and messagges
deliverd via dovecot-lda by MXs. After the upgrade from CentOS 6
to CentOS 7 I found the same issue as you (on dovecot-uidlist).</p>
<p>After many tests we decided to switch from LDA to LMTP, that was
already in our roadmap, so read and delivery of messagges is
always on the same backend. And the problem was solved.</p>
<p>I haven't found any others workarounds.</p>
<p>Swith from LDA to LMTP was not so simple for us since our MX
wasn't able to talk LMTP but we have write some custom C++ code
and was done. You should also consider to add some directors since
also incoming emails will transit from it.<br>
</p>
<p>If you would like to talk about how we solve on MXs side I will
happy to talk with you.</p>
<p>Ciao<br>
</p>
<div>Il 08/10/21 21:01, Jeremy Hanmer ha
scritto:<br>
</div></div><div>
<blockquote type="cite">
<div dir="ltr">
<div>I know this has been reported in the past, but I think I
have some useful new information on the problem. After an OS
upgrade from Ubuntu Xenial (4.4.0 kernel) to Ubuntu Focal
(5.4.0 kernel) and corresponding upgrade from Dovecot 2.2.27
to 2.3.7.2, we've started seeing broken uidlist files to an
extent that's making larger mail boxes nearly unusable because
the file is constantly being regenerated. I've also used the
2.3.16-2+ubuntu20.04 version distributed from <a href="http://dovecot.org" target="_blank">dovecot.org</a>
and the behavior is unchanged. The environment consists of NFS
mounts from a NetApp device, with a couple dozen MX servers
receiving mail and about a hundred IMAP/POP servers. <br>
</div>
<div><br>
</div>
<div>This is the exact error (note the blank after "invalid
data"):<br>
</div>
<div>Error: Mailbox INBOX: Broken file
/mnt/morty/morty2/gravest/x15775549/Maildir/dovecot-uidlist
line 373: Invalid data: <br>
</div>
<div><br>
</div>
<div>I've been able to trigger the problem rather easily by
piping an email to dovecot-lda in a loop and reading the
resulting dovecot-uidlist file on a different server. What it
shows is that occasionally we're seeing the last line of the
file prepended with a number of null bytes equal to the line
that's being written (for example, if the entry is "35322
:1633719038.M516419P3623238.pdx1-sub0-mail-mx202,S=2777,W=2832",
we'll have it prepended by 69 null bytes). This then breaks
the IMAP process' ability to read the file. My first thought
was to extend the retry functionality so the imap proces makes
more attempts to read the file when it detects a problem like
this, but would love input from someone more familiar with the
codebase.<br>
</div>
</div>
</blockquote>
<pre cols="72" style="font-family:monospace">--
Alessio Cecchi
Postmaster @ <a href="http://www.qboxmail.it" style="font-family:monospace" target="_blank">http://www.qboxmail.it</a>
<a href="https://www.linkedin.com/in/alessice" style="font-family:monospace" target="_blank">https://www.linkedin.com/in/alessice</a></pre>
</div>
</blockquote></div></div>