[Dovecot] Issues with VANISHED CHANGEDSINCE

[Michael J Rubinsky]

Quoting Timo Sirainen <tss at iki.fi>:

> On 6.11.2012, at 3.49, Michael J Rubinsky wrote:
>
>>> That would require infinitely storing the modseq of when each message
>>> was expunged. Not very nice. Also the RFC talks a lot about this
>>> situation. The SELECT command has two optional parameters to optimize
>>> it.
>>
>> The RFC *does* indicate that a server implementation could,  
>> strictly speaking, be considered in compliance without remembering  
>> modsequences for all expunged messages, but it does explicitly  
>> discourage such implementations. From RFC 5162 [4.1]:
>>
>>   Strictly speaking, a server implementation that doesn't remember mod-
>>   sequences associated with expunged messages can be considered
>>   compliant with this specification.  Such implementations return all
>>   expunged messages specified in the UID set of the UID FETCH
>>   (VANISHED) command every time, without paying attention to the
>>   specified CHANGEDSINCE mod-sequence.  Such implementations are
>>   discouraged, as they can end up returning VANISHED responses that are
>>   bigger than the result of a UID SEARCH command for the same UID set.
>
> This is talking about a server that doesn't permanently remember ANY  
> modseqs for expunges. Dovecot remembers them, not not infinitely.
>
>> It also gives advice to avoid infinitely storing the modsequences  
>> such as "expiring" sequences associated with older expunged  
>> messages, but assigning a single modsequence value to all of the  
>> expired expunged messages.
>
> Dovecot behaves as the section 4.3 describes. Note especially:
>
>    Note that indefinitely storing information about expunged messages
>    can cause storage and related problems for an implementation.
> ..
>    Hence, implementations are encouraged to adopt strategies to protect
>    against such storage problems, such as limiting the size of the queue
>    used to store mod-sequences for expunged messages and "expiring"
>    older records when this limit is reached.  When the selected
>    implementation-specific queue limit is reached, the oldest record(s)
>    are deleted from the queue (note that such records are located at the
>    queue head).  For all such "expired" records, the server needs to
>    store a single mod-sequence, which is the highest mod-sequence for
>    all "expired" expunged messages.
>
> This is exactly what Dovecot does. There is a single modseq  
> associated with all the previously expunged messages. If you try to  
> request expunges for that modseq, it returns all of the expunged  
> messages, which is what you're seeing as a problem.
>
> It might be helpful in some situations to decrease the precision and  
> remember for example:
>  * UIDs 1-100 were deleted with modseq 10 (in reality multiple times  
> between modseqs 1..10)
>  * UIDs 101-130 were deleted with modseq 15 (in reality with modseqs 11..15)
> .. and so on
>
> But this assumes that the expunged UID ranges compress well. If UIDs  
> are being deleted here and there it's still pretty wasteful to store  
> them.
>
> And yes, the current way may be forgetting them a bit too early when  
> a lot of other unrelated changes are happening. It would be possible  
> to keep a separate expunge log which could remember the expunges  
> longer. But that would be yet another different index file for  
> Dovecot, which annoyingly complicates everything. And currently  
> since it sounds like the only problem is activesync implementation  
> using it, I'm not very interested in spending a lot of time on it.

> These defines in mail-transaction-log-private.h anyway can be  
> changed to make it much less likely to see your problem:
>
> /* Rotate when log is older than ROTATE_TIME and larger than MIN_SIZE */
> #define MAIL_TRANSACTION_LOG_ROTATE_MIN_SIZE (1024*32)
> /* If log is larger than MAX_SIZE, rotate regardless of the time */
> #define MAIL_TRANSACTION_LOG_ROTATE_MAX_SIZE (1024*1024)
> #define MAIL_TRANSACTION_LOG_ROTATE_TIME (60*5)
>
> /* Delete .log.2 files older than this many seconds. Don't be too eager,
>    older files are useful for QRESYNC and dsync. */
> #define MAIL_TRANSACTION_LOG2_STALE_SECS (60*60*24*2)
>
> Maybe the defaults could be changed..

Thanks for the information and clarification, Timo.


-- 
mike

The Horde Project (www.horde.org)
mrubinsk at horde.org
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 6062 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://dovecot.org/pipermail/dovecot/attachments/20121106/792f95ca/attachment-0004.bin>