[Dovecot] Issues with VANISHED CHANGEDSINCE

[Michael M Slusarz]

Quoting Timo Sirainen <tss at iki.fi>:

> On 8.11.2012, at 0.34, Michael M Slusarz wrote:
>
>> Quoting Michael M Slusarz <slusarz at curecanti.org>:
>>
>>> I see your point, but the problem is that is not intuitive when  
>>> reading the RFC.  One part of the RFC defines the behavior of  
>>> VANISHED (EARLIER) as only returning changes since the  
>>> mod-sequence given.  And you are correct that another part of the  
>>> RFC says that, essentially, a server is allowed to break this  
>>> required response.
>>>
>>> I'm thinking that this is more of an issue with the way the RFC is  
>>> written.  I'll move this over to the imap protocol list to get  
>>> further input.
>>
>> Sigh.  Never mind.  For some reason, I completely ignored (missed?)  
>> this part of the RFC:
>>
>>   Note: A server that receives a mod-sequence smaller than <minmodseq>,
>>   where <minmodseq> is the value of the smallest expunged mod-sequence
>>   it remembers minus one, MUST behave as if it was requested to report
>>   all expunged messages from the provided UID set parameter.
>>
>> So you are right, I was wrong, and the world is good.
>
> I wonder how much would it help if you
>
> a) Used the uidset/seqset parameters with SELECT command

We *do* use this information. However, this is not (necessarily)  
useful for the activesync query that was the genesis of this thread. A  
bit of background on our MUA design is necessary.

For Horde/IMP, all IMAP server configuration is done through the IMP  
application. As part of this configuration, a cache backend can be  
configured.

There are multiple potential users of this IMAP object.  Within IMP  
itself, multiple sessions can be open at any one time.  Additionally,  
several views of IMP, our dynamic view and our smartmobile view, have  
another cache of messages kept on the browser side.  Finally, the  
ActiveSync library also uses the IMAP object configured by IMP.

Anytime the IMAP object is accessed, we are syncing the mailbox with  
the IMP-configured cache.  For QRESYNC, we use the SELECT/EXAMINE  
uidset parameter.

The problem is that any particular view may not be sync'd to the same  
state as the IMP cache.  For example, if someone is using the web  
application and their phone is syncing via ActiveSync, it is quite  
likely that the activesync cached mod-sequence value will NOT equal  
the IMP cached mod-sequence value.  So this is when explicitly calling  
FETCH VANISHED CHANGEDSINCE is needed.

The good news: once we get the CHANGEDSINCE FETCH information, we  
don't need to do a separate flags sync since this information has  
already been cached within the IMAP object (via either the  
CHANGEDSINCE call or, more likely, a previous FETCH call in another  
session).  Further optimization: in the case where the original  
QRESYNC/CONDSTORE sync matches the mod-sequence of whatever  
object/view is accessing the IMAP object, which should be the most  
common occurrence, there is no need to perform any additional  
FETCH/SEARCH calls since we cache the results of the initial mailbox  
sync and return this data.

Might be a long-winded explanation, but just wanted to show why FETCH  
VANISHED CHANGEDSINCE MUST be used by a client even if taking  
advantage of QRESYNC SELECT/EXAMINE syncing.  In other words - I'd  
like to think that my imap implementation is not broken :)

> b) Dovecot implemented it slightly better than required by RFC:
> http://www.ietf.org/mail-archive/web/lemonade/current/msg04771.html

I spent a week or so trying to cache message sequence number -> UID  
mapping.  And determined it was more trouble than it was worth.

The gains from more compact VANISHED responses in SELECT/EXAMINE are  
minimal compared to the expense to track them.  And the only other  
reason for tracking - the possibility that EXPUNGEs return EXPUNGED  
responses instead of VANISHED if the UIDs of the actually expunged  
messages are needed - can be worked around by doing a UID SEARCH call  
after the EXPUNGE is over and comparing to the list of UIDs that were  
given to UID EXPUNGE (with the further optimization that I cache  
MSN->UID while in a mailbox, which should catch the "STORE  
(\Deleted)/EXPUNGE" common when using a Trash mailbox or immediate  
message deletion).  Unless I am missing something else that MSNs are  
necessary?

michael