Dovecot 2.2.16: disappearing messages, mismatched summaries, duplicated messages, excessive full re-downloads

[David Gessel]

-------- Original Message --------
Subject: Re: Dovecot 2.2.16: disappearing messages, mismatched summaries, duplicated messages, excessive full re-downloads
From: Timo Sirainen <tss at iki.fi>
To: David Gessel <gessel at blackrosetech.com>
Date: Thu Apr 16 2015 21:31:30 GMT+0300 (Arabic Standard Time)

> On 16 Apr 2015, at 04:45, David Gessel <gessel at blackrosetech.com> wrote:
>>
>> My Dovecot 2.2.16 server is interacting poorly with a variety of mail clients.  The symptoms include disappearing messages, mismatched summaries, duplicated messages, and excessive full re-downloads.
> 
> What were you running before Dovecot v2.2.16? An earlier Dovecot? A whole different system?
> 
>> dovecot.log shows runs of this error:
>> Apr 15 09:02:37 imap(user at domain.com): Warning: Maildir /mail/domain.com/user//.Sent: Expunged message reappeared, giving a new UID (old uid=306286, file=1429102024.M826331P1156.host,S=1678,W=1724:2,S)
>>
>> or this error:
>> Apr 15 10:04:00 indexer-worker(user at domain.com): Warning: Maildir /mail/domain.com/user//.Sent: Expunged message reappeared, giving a new UID (old uid=306803, file=1428785463.M645546P27215.host,S=5389,W=5442:2,S)
> ..
>> The first entry in dovecot.log.0 is at Apr 15 10:00:12, and the last entry at Apr 15 12:00:13; Dovecot is logging 4,490 expunged message reappeared messages per hour.  Note the server has 4 accounts set up and very limited traffic, so 4,490 per hour would seem like a lot of anything to be logged.
> 
> These really shouldn't be happening..
> 

One possibility is that there is a scrub/verify routine running that is checking the actual size vs. reported size of messages, and perhaps that routine doesn't know about ZFS compression:

zroot/ezjail                             used           117G      -
zroot/ezjail                             compressratio  1.25x     -
zroot/ezjail                             compression    lz4       local
zroot/ezjail                             logicalused    137G      -

and is seeing something anomalous and trying to "fix" that, triggering a rescan.  A long shot and seems unlikely, but I have no other ideas.

I also note from the documentation on the wiki:

"Because Dovecot uses its own non-standard locking (dovecot-uidlist.lock dotlock file), other MUAs accessing the maildir don't support it. This means that if another MUA is updating messages' flags or expunging messages, Dovecot might temporarily lose some message. After the next sync when it finds it again, an error message may be written to log and the message will receive a new UID. "

...which are the right symptoms, but seems unlikely to be the right cause.

And that:

"Maildir format is somewhat compatible with MH format. This is sometimes a problem when people configure their procmail to deliver mails to Maildir/new. This makes procmail create the messages in MH format, which basically means that the file is called msg.inode_number. While this appears to work first, after expunging messages from the maildir the inodes are freed and will be reused later. This means that another file with the same name may come to the maildir, which makes Dovecot think that an expunged file reappeared into the mailbox and an error is logged. "

...which is exactly the error, but the 

And since I'm using Dovecot LMTP/LDA, it seems odd that errors would be generated. 

My main.cf config includes:

virtual_transport = dovecot
dovecot_destination_recipient_limit = 1
home_mailbox = Maildir/
mail_spool_directory = /mail
mailbox_command = /usr/local/libexec/dovecot/deliver
mailbox_transport = dovecot