Dovecot 2.2.16: disappearing messages, mismatched summaries, duplicated messages, excessive full re-downloads

[David Gessel]

The update to dovecot2-2.2.18/dovecot-pigeonhole-0.4.8 seems to have made problems worse.  Whole directories don't load at all - or they may load a weird segment - like mail from a few months ago back and mail from the last two days, but nothing in between.

I did a 
# find /mail -name "dovecot.index*" -type f -delete

and then on the client side (K9) things clear up, on TB a Right click on folder -> properties -> repair folder cleans things up.  Of course all messages have to be redownloaded so it isn't exactly a performant option.  

Is there a way to completely disable caching on dovecot?  I know it is sort of the raison d'être, but if I can't solve the problems with dovecot, I'll have to try a different, non-caching/indexing server anyway and at least it would help narrow down the issues.


-------- Original Message --------
Subject: Re: Dovecot 2.2.16: disappearing messages, mismatched summaries, duplicated messages, excessive full re-downloads
From: David Gessel <gessel at blackrosetech.com>
To: Timo Sirainen <tss at iki.fi>
Date: Thu Apr 23 2015 18:39:02 GMT+0400 (Arabian Standard Time)

> very_dirty_syncs helped a lot - the problems still happen but only about every 30 minutes.  It seems this flag prevents re-indexing on whatever was constantly inducing it (with every client-check, if there was a change written to the directory, is my best guess watching the logs go).
> 
> I still get all the same symptoms, but far less frequently.
> 
> I'm inclined to believe, as trivial as it may be to enumerate, that:
> 
> Something is triggering dovecot to believe the indexes need to be rebuilt.
> 
> When checking mail during the rebuild, clients get confused by UIDs in transition.
> 
> Today I had to quit TB and restart because duplicates filled my preview pane.  Mulberry has been better, when it gets confused now, it is usually sufficient to initiate a manual check or expunge to get the list to update.
> 
> I would think that sdbox would alleviate these issues, no?
> 
> -------- Original Message --------
> Subject: Re: Dovecot 2.2.16: disappearing messages, mismatched summaries, duplicated messages, excessive full re-downloads
> From: Timo Sirainen <tss at iki.fi>
> To: David Gessel <gessel at blackrosetech.com>
> Date: Sat Apr 18 2015 17:38:02 GMT+0300 (Arabic Standard Time)
> 
>> On 18 Apr 2015, at 16:00, David Gessel <gessel at blackrosetech.com> wrote:
>>>
>>>> No. My best guess is that (your) ZFS+FreeBSD is simply not behaving the way Dovecot wants filesystems to behave (readdir() sometimes skipping over files). I don't have any other suggestion than trying a different filesystem or OS. I haven't heard ZFS having these problems before though.
>>>
>>>
>>> Or migrating to sdbox.  Absent any further suggestions, that will be my next step.  My commitment to Maildir is far lower than my commitment to FreeBSD/ZFS.  It is just a historical preference coming from a decade+ of courier IMAP. 
>>
>> Actually as a workaround you could try maildir_very_dirty_syncs=yes which gets rid of most of the readdir()s.
>>