[Dovecot] Issues with VANISHED CHANGEDSINCE

[Timo Sirainen]

On 6.11.2012, at 3.49, Michael J Rubinsky wrote:

>> That would require infinitely storing the modseq of when each message
>> was expunged. Not very nice. Also the RFC talks a lot about this
>> situation. The SELECT command has two optional parameters to optimize
>> it.
> 
> The RFC *does* indicate that a server implementation could, strictly speaking, be considered in compliance without remembering modsequences for all expunged messages, but it does explicitly discourage such implementations. From RFC 5162 [4.1]:
> 
>   Strictly speaking, a server implementation that doesn't remember mod-
>   sequences associated with expunged messages can be considered
>   compliant with this specification.  Such implementations return all
>   expunged messages specified in the UID set of the UID FETCH
>   (VANISHED) command every time, without paying attention to the
>   specified CHANGEDSINCE mod-sequence.  Such implementations are
>   discouraged, as they can end up returning VANISHED responses that are
>   bigger than the result of a UID SEARCH command for the same UID set.

This is talking about a server that doesn't permanently remember ANY modseqs for expunges. Dovecot remembers them, not not infinitely.

> It also gives advice to avoid infinitely storing the modsequences such as "expiring" sequences associated with older expunged messages, but assigning a single modsequence value to all of the expired expunged messages.

Dovecot behaves as the section 4.3 describes. Note especially:

   Note that indefinitely storing information about expunged messages
   can cause storage and related problems for an implementation.
..
   Hence, implementations are encouraged to adopt strategies to protect
   against such storage problems, such as limiting the size of the queue
   used to store mod-sequences for expunged messages and "expiring"
   older records when this limit is reached.  When the selected
   implementation-specific queue limit is reached, the oldest record(s)
   are deleted from the queue (note that such records are located at the
   queue head).  For all such "expired" records, the server needs to
   store a single mod-sequence, which is the highest mod-sequence for
   all "expired" expunged messages.

This is exactly what Dovecot does. There is a single modseq associated with all the previously expunged messages. If you try to request expunges for that modseq, it returns all of the expunged messages, which is what you're seeing as a problem.

It might be helpful in some situations to decrease the precision and remember for example:
 * UIDs 1-100 were deleted with modseq 10 (in reality multiple times between modseqs 1..10)
 * UIDs 101-130 were deleted with modseq 15 (in reality with modseqs 11..15)
.. and so on

But this assumes that the expunged UID ranges compress well. If UIDs are being deleted here and there it's still pretty wasteful to store them.

And yes, the current way may be forgetting them a bit too early when a lot of other unrelated changes are happening. It would be possible to keep a separate expunge log which could remember the expunges longer. But that would be yet another different index file for Dovecot, which annoyingly complicates everything. And currently since it sounds like the only problem is activesync implementation using it, I'm not very interested in spending a lot of time on it.

These defines in mail-transaction-log-private.h anyway can be changed to make it much less likely to see your problem:

/* Rotate when log is older than ROTATE_TIME and larger than MIN_SIZE */
#define MAIL_TRANSACTION_LOG_ROTATE_MIN_SIZE (1024*32)
/* If log is larger than MAX_SIZE, rotate regardless of the time */
#define MAIL_TRANSACTION_LOG_ROTATE_MAX_SIZE (1024*1024)
#define MAIL_TRANSACTION_LOG_ROTATE_TIME (60*5)

/* Delete .log.2 files older than this many seconds. Don't be too eager,
   older files are useful for QRESYNC and dsync. */
#define MAIL_TRANSACTION_LOG2_STALE_SECS (60*60*24*2)

Maybe the defaults could be changed..