Dovecot 2.2.16: disappearing messages, mismatched summaries, duplicated messages, excessive full re-downloads

[David Gessel]

Steffen,

I apologize for missing your helpful message - I wasn't intentionally ignoring it. One of the artifacts of my mail issues is that mail disappears, this included.  

-------- Original Message --------
Subject: Re: Dovecot 2.2.16: disappearing messages, mismatched summaries, duplicated messages, excessive full re-downloads
From: Steffen Kaiser <skdovecot at smail.inf.fh-brs.de>
To: David Gessel <gessel at blackrosetech.com>
Date: Fri Apr 24 2015 10:29:46 GMT+0300 (Arabic Standard Time)

> On Thu, 23 Apr 2015, David Gessel wrote:
> 
>> I'm inclined to believe, as trivial as it may be to enumerate, that:
> 
>> Something is triggering dovecot to believe the indexes need to be rebuilt.
>> When checking mail during the rebuild, clients get confused by UIDs in transition.
> 
>> I would think that sdbox would alleviate these issues, no?
> 
> The real problem is that you do not know _why_ "Something is triggering dovecot to believe the indexes need to be rebuilt".

I'm not 100% sure that's what is happening.

> 
> This is the same for sdbox and mdbox, IMHO.

that would be sad - but I haven't tried that yet.

> 
> That's why I asked about if some external process is trying to change the mail storage. Is there something except Dovecot that changes the mtime of the directories "new", "cur" or Maildir base?

Not that I am aware of.  this is a jail only running mail.

> Do you deliver messages without Dovecot LDA/LMTP? 

Dovecot LDA only.

>Do you store different information in the Maildir? 

nothing not part of Dovecot's processes - but the full text index files are also stored there.

> Do you (not) have separate mail storage and user home directories? 

Virtual mail configuration - home directories are completely isolated.  Nothing happens in /mail that isn't mail related.

> Do you run a virus checker on file system level?

no, but amavisd calls clamAV on inbound messages.

> 
> Do you run two Dovecot instances on the same server, maybe as left over from some testing or crash?

certainly not intentionally and I'm pretty confident not actually; these issues survive many reboots without much variation, so not from testing.

> 
> -- Steffen Kaiser


Some updates to the process:

I experimented with all variations of Mail processes values with no real improvement - some perhaps but likely as not just observational variations rather than meaningful data.  Specifically in 10-mail.conf:
 mmap_disable = yes
 mail_fsync = always
 lock_method = flock  

I'm probably suffering from confirmation bias, but I think things got a little bit better.

However, I still got double messages in TB and K9 on some checks.  Then I looked at the IMAP capability string returned from telnet localhost 993.  It didn't include NAMESPACE.  The post-login enumeration did, but not the pre-login.  

adding
imap_capability = +NAMESPACE
to 20-imap.conf
seems to have cleared up the appearance of double entries in clients.  I had one message's header/display get confused (another symptom of the issues) but given the problems my local client database is pretty scrambled.  when I have a decent network connection for a few days, I'll try recreating my client database and see if that helps.