[Dovecot] Issues with VANISHED CHANGEDSINCE

[Michael M Slusarz]

Quoting Michael J Rubinsky <mrubinsk at horde.org>:

>>>> I don't think they become incorrect, just that there are more of  
>>>> them than really necessary? Yes, there's a tipping point. It's  
>>>> when the modseq no longer exists in the dovecot.index.log* files,  
>>>> which get rotated once in a while. This shouldn't happen very  
>>>> often.
>>>
>>> FYI, I see this about once every two weeks, sometimes more  
>>> frequently. Is there anything we can do to reduce the chance of  
>>> this happening?
>>
>> How do you see it? Does it break something? Why?
>>
>> You can change it only by increasing the dovecot.index.log sizes,  
>> which requires changing the code.
>
> It breaks ActiveSync synchronization of mailboxes. When this  
> happens, the sync state of the mailbox needs to be reset, which  
> causes the entire mailbox to be resynchronized to the mobile device.  
> This can lead to a not-insignificant amount of wasted bandwidth and  
> battery power for the device. There have been times when this has  
> happened multiple times in a single day.
>
> Not resetting the state leads to multiple issues on the device due  
> to sending it thousands of deletion commands for messages it knows  
> nothing about.

My argument is much simpler: it is blatantly breaking the RFC.  From  
RFC 5162 [3.2]:

    The VANISHED UID FETCH modifier instructs the server to report those
    messages from the UID set parameter that have been expunged and whose
    associated mod-sequence is larger than the specified mod-sequence.
    **That is, the client requests to be informed of messages from the
    specified set that were expunged since the specified  
mod-sequence.** (emphasis added)

If you are including UIDs in the FETCH return that have NOT been  
expunged since the given mod-sequence, that directly contradicts this  
language.  The clear intent of VANISHED UID FETCH is to provide the  
list of messages that existed in the mailbox at mod-sequence and no  
longer exist in the mailbox as of the current HIGHESTMODSEQ.

As Mike R. has demonstrated, it is plausible that an MUA can only  
provide the MODSEQ of its cache state and has no knowledge of the UIDs  
it has actually cached.  So having to parse through a (potentially)  
giant list of UIDs can be a performance killer (imagine the wasted  
bandwidth of having to upload a million UIDs to a phone every time you  
sync).

michael