[Dovecot] Issues with VANISHED CHANGEDSINCE
I've been seeing the following wonky behavior with Dovecot. Currently
this is with Dovecot 2.0.19, but I was also seeing it in earlier
versions as well, including versions from the 1.x series. Some
background, this is from Horde's ActiveSync library, when it is trying
to determine what UIDs have recently been expunged. I can guarantee
that when this command was sent, there was only one or two emails that
were removed since the provided MODSEQ value.
C: 8 UID FETCH 1:* UID (VANISHED CHANGEDSINCE 587742)
S: * VANISHED (EARLIER)
1:119944,119947:123482,123484:128043,128045:130388,130390:130456,130458:132127,132129:135832,135834:136201,136203:136301,136303:136316,136318:138502,138504:140099,140101:141908,141910:142607,142609:142744,142746:142865,142867:143160,143164:143173,143176:143179,143181:143187,143189:143190,143192:143195,143197:143270,143272:143273,143275:143280,143282:143302,143305:143307,143309:143318,143320:143333,143335:143337,143339:143341,143343:143375,143377:143400,143402:143551,143553:143637,143640:143706,143708:144069,144071:144178,144180:144307,144309:144316,144318:144329,144331:144931,144933:144983,144985:145000,145002:145069,145071:145072,145074:145132,145136:145178,145180:145191,145193
Also, to verify it wasn't something screwy with my server, Michael
Slusarz provided me with this from his server:
<quote> There's definitely something wonky going on in the code. There's a certain tipping point of modseqs where the values become incorrect. For a mailbox with HIGHESTMODSEQ of 54004, I see this:
a uid fetch 1:* UID (VANISHED CHANGEDSINCE 53881)
- VANISHED (EARLIER) 1:37308,37310:40788,40791:41032,41034:41083 a OK Fetch completed. b uid fetch 1:* UID (VANISHED CHANGEDSINCE 53882)
- VANISHED (EARLIER) 41029:41030,41047:41083 b OK Fetch completed. </quote>
Thanks for any guidance.
mike
The Horde Project (www.horde.org) mrubinsk@horde.org
Quoting Michael J Rubinsky mrubinsk@horde.org:
Also, to verify it wasn't something screwy with my server, Michael
Slusarz provided me with this from his server:<quote> There's definitely something wonky going on in the code. There's a certain tipping point of modseqs where the values become incorrect. For a mailbox with HIGHESTMODSEQ of 54004, I see this:
a uid fetch 1:* UID (VANISHED CHANGEDSINCE 53881)
- VANISHED (EARLIER) 1:37308,37310:40788,40791:41032,41034:41083 a OK Fetch completed. b uid fetch 1:* UID (VANISHED CHANGEDSINCE 53882)
- VANISHED (EARLIER) 41029:41030,41047:41083 b OK Fetch completed. </quote>
FYI, this is output from Dovecot 2.1.10.
michael
On 5.11.2012, at 18.13, Michael J Rubinsky wrote:
I've been seeing the following wonky behavior with Dovecot. Currently this is with Dovecot 2.0.19, but I was also seeing it in earlier versions as well, including versions from the 1.x series. Some background, this is from Horde's ActiveSync library, when it is trying to determine what UIDs have recently been expunged. I can guarantee that when this command was sent, there was only one or two emails that were removed since the provided MODSEQ value. .. <quote> There's definitely something wonky going on in the code. There's a certain tipping point of modseqs where the values become incorrect.
I don't think they become incorrect, just that there are more of them than really necessary? Yes, there's a tipping point. It's when the modseq no longer exists in the dovecot.index.log* files, which get rotated once in a while. This shouldn't happen very often.
For a mailbox with HIGHESTMODSEQ of 54004, I see this:
a uid fetch 1:* UID (VANISHED CHANGEDSINCE 53881)
- VANISHED (EARLIER) 1:37308,37310:40788,40791:41032,41034:41083 a OK Fetch completed. b uid fetch 1:* UID (VANISHED CHANGEDSINCE 53882)
- VANISHED (EARLIER) 41029:41030,41047:41083 b OK Fetch completed. </quote>
So in b) case Dovecot read the dovecot.index.log and returned the expunges found from there. In a) case it didn't know what was expunged since the modseq, so it returned UIDs of all the expunged mails.
Quoting Timo Sirainen tss@iki.fi:
On 5.11.2012, at 18.13, Michael J Rubinsky wrote:
I've been seeing the following wonky behavior with Dovecot.
Currently this is with Dovecot 2.0.19, but I was also seeing it in
earlier versions as well, including versions from the 1.x series.
Some background, this is from Horde's ActiveSync library, when it
is trying to determine what UIDs have recently been expunged. I
can guarantee that when this command was sent, there was only one
or two emails that were removed since the provided MODSEQ value. .. <quote> There's definitely something wonky going on in the code. There's a
certain tipping point of modseqs where the values become incorrect.I don't think they become incorrect, just that there are more of
them than really necessary? Yes, there's a tipping point. It's when
the modseq no longer exists in the dovecot.index.log* files, which
get rotated once in a while. This shouldn't happen very often.
FYI, I see this about once every two weeks, sometimes more frequently.
Is there anything we can do to reduce the chance of this happening?
-- mike
The Horde Project (www.horde.org) mrubinsk@horde.org
On 5.11.2012, at 20.37, Michael J Rubinsky wrote:
On 5.11.2012, at 18.13, Michael J Rubinsky wrote:
I've been seeing the following wonky behavior with Dovecot. Currently this is with Dovecot 2.0.19, but I was also seeing it in earlier versions as well, including versions from the 1.x series. Some background, this is from Horde's ActiveSync library, when it is trying to determine what UIDs have recently been expunged. I can guarantee that when this command was sent, there was only one or two emails that were removed since the provided MODSEQ value. .. <quote> There's definitely something wonky going on in the code. There's a certain tipping point of modseqs where the values become incorrect.
I don't think they become incorrect, just that there are more of them than really necessary? Yes, there's a tipping point. It's when the modseq no longer exists in the dovecot.index.log* files, which get rotated once in a while. This shouldn't happen very often.
FYI, I see this about once every two weeks, sometimes more frequently. Is there anything we can do to reduce the chance of this happening?
How do you see it? Does it break something? Why?
You can change it only by increasing the dovecot.index.log sizes, which requires changing the code.
Quoting Timo Sirainen tss@iki.fi:
On 5.11.2012, at 20.37, Michael J Rubinsky wrote:
On 5.11.2012, at 18.13, Michael J Rubinsky wrote:
I've been seeing the following wonky behavior with Dovecot.
Currently this is with Dovecot 2.0.19, but I was also seeing it
in earlier versions as well, including versions from the 1.x
series. Some background, this is from Horde's ActiveSync
library, when it is trying to determine what UIDs have recently
been expunged. I can guarantee that when this command was sent,
there was only one or two emails that were removed since the
provided MODSEQ value. .. <quote> There's definitely something wonky going on in the code. There's
a certain tipping point of modseqs where the values become
incorrect.I don't think they become incorrect, just that there are more of
them than really necessary? Yes, there's a tipping point. It's
when the modseq no longer exists in the dovecot.index.log* files,
which get rotated once in a while. This shouldn't happen very often.FYI, I see this about once every two weeks, sometimes more
frequently. Is there anything we can do to reduce the chance of
this happening?How do you see it? Does it break something? Why?
You can change it only by increasing the dovecot.index.log sizes,
which requires changing the code.
It breaks ActiveSync synchronization of mailboxes. When this happens,
the sync state of the mailbox needs to be reset, which causes the
entire mailbox to be resynchronized to the mobile device. This can
lead to a not-insignificant amount of wasted bandwidth and battery
power for the device. There have been times when this has happened
multiple times in a single day.
Not resetting the state leads to multiple issues on the device due to
sending it thousands of deletion commands for messages it knows
nothing about.
mike
The Horde Project (www.horde.org) mrubinsk@horde.org
Quoting Michael J Rubinsky mrubinsk@horde.org:
I don't think they become incorrect, just that there are more of
them than really necessary? Yes, there's a tipping point. It's
when the modseq no longer exists in the dovecot.index.log* files,
which get rotated once in a while. This shouldn't happen very
often.FYI, I see this about once every two weeks, sometimes more
frequently. Is there anything we can do to reduce the chance of
this happening?How do you see it? Does it break something? Why?
You can change it only by increasing the dovecot.index.log sizes,
which requires changing the code.It breaks ActiveSync synchronization of mailboxes. When this
happens, the sync state of the mailbox needs to be reset, which
causes the entire mailbox to be resynchronized to the mobile device.
This can lead to a not-insignificant amount of wasted bandwidth and
battery power for the device. There have been times when this has
happened multiple times in a single day.Not resetting the state leads to multiple issues on the device due
to sending it thousands of deletion commands for messages it knows
nothing about.
My argument is much simpler: it is blatantly breaking the RFC. From
RFC 5162 [3.2]:
The VANISHED UID FETCH modifier instructs the server to report those
messages from the UID set parameter that have been expunged and whose
associated mod-sequence is larger than the specified mod-sequence.
**That is, the client requests to be informed of messages from the
specified set that were expunged since the specified mod-sequence.** (emphasis added)
If you are including UIDs in the FETCH return that have NOT been
expunged since the given mod-sequence, that directly contradicts this
language. The clear intent of VANISHED UID FETCH is to provide the
list of messages that existed in the mailbox at mod-sequence and no
longer exist in the mailbox as of the current HIGHESTMODSEQ.
As Mike R. has demonstrated, it is plausible that an MUA can only
provide the MODSEQ of its cache state and has no knowledge of the UIDs
it has actually cached. So having to parse through a (potentially)
giant list of UIDs can be a performance killer (imagine the wasted
bandwidth of having to upload a million UIDs to a phone every time you
sync).
michael
On Mon, 2012-11-05 at 12:43 -0700, Michael M Slusarz wrote:
My argument is much simpler: it is blatantly breaking the RFC. From
RFC 5162 [3.2]:The VANISHED UID FETCH modifier instructs the server to report those messages from the UID set parameter that have been expunged and whose associated mod-sequence is larger than the specified mod-sequence. **That is, the client requests to be informed of messages from the specified set that were expunged since the specifiedmod-sequence.** (emphasis added)
If you are including UIDs in the FETCH return that have NOT been
expunged since the given mod-sequence, that directly contradicts this
language. The clear intent of VANISHED UID FETCH is to provide the
list of messages that existed in the mailbox at mod-sequence and no
longer exist in the mailbox as of the current HIGHESTMODSEQ.
That would require infinitely storing the modseq of when each message was expunged. Not very nice. Also the RFC talks a lot about this situation. The SELECT command has two optional parameters to optimize it.
Quoting Timo Sirainen tss@iki.fi:
On Mon, 2012-11-05 at 12:43 -0700, Michael M Slusarz wrote:
My argument is much simpler: it is blatantly breaking the RFC. From RFC 5162 [3.2]:
The VANISHED UID FETCH modifier instructs the server to report those messages from the UID set parameter that have been expunged and whose associated mod-sequence is larger than the specified mod-sequence. **That is, the client requests to be informed of messages from the specified set that were expunged since the specifiedmod-sequence.** (emphasis added)
If you are including UIDs in the FETCH return that have NOT been expunged since the given mod-sequence, that directly contradicts this language. The clear intent of VANISHED UID FETCH is to provide the list of messages that existed in the mailbox at mod-sequence and no longer exist in the mailbox as of the current HIGHESTMODSEQ.
That would require infinitely storing the modseq of when each message was expunged. Not very nice. Also the RFC talks a lot about this situation. The SELECT command has two optional parameters to optimize it.
The RFC *does* indicate that a server implementation could, strictly
speaking, be considered in compliance without remembering modsequences
for all expunged messages, but it does explicitly discourage such
implementations. From RFC 5162 [4.1]:
Strictly speaking, a server implementation that doesn't remember mod-
sequences associated with expunged messages can be considered
compliant with this specification. Such implementations return all
expunged messages specified in the UID set of the UID FETCH
(VANISHED) command every time, without paying attention to the
specified CHANGEDSINCE mod-sequence. Such implementations are
discouraged, as they can end up returning VANISHED responses that are
bigger than the result of a UID SEARCH command for the same UID set.
Clients that use the message sequence match data can reduce the scope
of this VANISHED response substantially in the typical case where
expunges have not happened, or happen only toward the end of the
mailbox.It also gives advice to avoid infinitely storing the modsequences such
as "expiring" sequences associated with older expunged messages, but
assigning a single modsequence value to all of the expired expunged
messages.
-- mike
The Horde Project (www.horde.org) mrubinsk@horde.org
On 6.11.2012, at 3.49, Michael J Rubinsky wrote:
That would require infinitely storing the modseq of when each message was expunged. Not very nice. Also the RFC talks a lot about this situation. The SELECT command has two optional parameters to optimize it.
The RFC *does* indicate that a server implementation could, strictly speaking, be considered in compliance without remembering modsequences for all expunged messages, but it does explicitly discourage such implementations. From RFC 5162 [4.1]:
Strictly speaking, a server implementation that doesn't remember mod- sequences associated with expunged messages can be considered compliant with this specification. Such implementations return all expunged messages specified in the UID set of the UID FETCH (VANISHED) command every time, without paying attention to the specified CHANGEDSINCE mod-sequence. Such implementations are discouraged, as they can end up returning VANISHED responses that are bigger than the result of a UID SEARCH command for the same UID set.
This is talking about a server that doesn't permanently remember ANY modseqs for expunges. Dovecot remembers them, not not infinitely.
It also gives advice to avoid infinitely storing the modsequences such as "expiring" sequences associated with older expunged messages, but assigning a single modsequence value to all of the expired expunged messages.
Dovecot behaves as the section 4.3 describes. Note especially:
Note that indefinitely storing information about expunged messages can cause storage and related problems for an implementation. .. Hence, implementations are encouraged to adopt strategies to protect against such storage problems, such as limiting the size of the queue used to store mod-sequences for expunged messages and "expiring" older records when this limit is reached. When the selected implementation-specific queue limit is reached, the oldest record(s) are deleted from the queue (note that such records are located at the queue head). For all such "expired" records, the server needs to store a single mod-sequence, which is the highest mod-sequence for all "expired" expunged messages.
This is exactly what Dovecot does. There is a single modseq associated with all the previously expunged messages. If you try to request expunges for that modseq, it returns all of the expunged messages, which is what you're seeing as a problem.
It might be helpful in some situations to decrease the precision and remember for example:
- UIDs 1-100 were deleted with modseq 10 (in reality multiple times between modseqs 1..10)
- UIDs 101-130 were deleted with modseq 15 (in reality with modseqs 11..15) .. and so on
But this assumes that the expunged UID ranges compress well. If UIDs are being deleted here and there it's still pretty wasteful to store them.
And yes, the current way may be forgetting them a bit too early when a lot of other unrelated changes are happening. It would be possible to keep a separate expunge log which could remember the expunges longer. But that would be yet another different index file for Dovecot, which annoyingly complicates everything. And currently since it sounds like the only problem is activesync implementation using it, I'm not very interested in spending a lot of time on it.
These defines in mail-transaction-log-private.h anyway can be changed to make it much less likely to see your problem:
/* Rotate when log is older than ROTATE_TIME and larger than MIN_SIZE */ #define MAIL_TRANSACTION_LOG_ROTATE_MIN_SIZE (1024*32) /* If log is larger than MAX_SIZE, rotate regardless of the time */ #define MAIL_TRANSACTION_LOG_ROTATE_MAX_SIZE (1024*1024) #define MAIL_TRANSACTION_LOG_ROTATE_TIME (60*5)
/* Delete .log.2 files older than this many seconds. Don't be too eager, older files are useful for QRESYNC and dsync. */ #define MAIL_TRANSACTION_LOG2_STALE_SECS (60*60*24*2)
Maybe the defaults could be changed..
Am 06.11.2012 13:17, schrieb Timo Sirainen:
the only problem is activesync implementation using it
Hi, Michael, as i am going to implement horde 5 active sync server in near Future with dovecot , i followed this in high interest
I know horde active sync server is different to z-push so sorry my question, i recent use z-push with dovecot and never noticed any problems with it, so how does z-push handle this "problem" comparing horde, any chance to workaround it in horde code ?
Hacking dovecot might be complex, as it should fit in so many stuff and setups
Anyway i agree many people using active sync in their smartphone these days and new outlook will also have a active sync mode, so there shouldnt stay any fixable problems ( where ever they may be ), but in my understanding the imap server ( dovecot ) acts more like a backhand here did you test against recent versions of cyrus etc, how is their behave in this case ?
Best Regards MfG Robert Schetterer
-- [*] sys4 AG
http://sys4.de, +49 (89) 30 90 46 64 Franziskanerstraße 15, 81669 München
Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263 Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer Aufsichtsratsvorsitzender: Joerg Heidrich
Quoting Robert Schetterer rs@sys4.de:
Am 06.11.2012 13:17, schrieb Timo Sirainen:
the only problem is activesync implementation using it
Hi, Michael, as i am going to implement horde 5 active sync server in near Future with dovecot , i followed this in high interest
I know horde active sync server is different to z-push so sorry my question, i recent use z-push with dovecot and never noticed any problems with it, so how does z-push handle this "problem" comparing horde, any chance to workaround it in horde code ?
ZPush's default "imap backend" implementation uses PHP's imap
implementation, which has _many_ problems. Horde's uses
Horde_Imap_Client a PHP native implementation with tons of
improvements over just about any other available IMAP client
implementation in PHP.
The problem I'm describing stems from our ActiveSync code trying to
take advantage of IMAP extensions such as QRESYNC, where the server
can track and report things like expunged messages, flag changes
etc... since a known point without us having to cache and track every
bit of information about the device's state. This is what ZPush does,
it uses plaintext files on the server and writes out all the message
ids and flag states that were sent to each device. Then during every
sync, it retrieves the complete list of message ids and flag state
from the IMAP server again and compares them against the data in the
text file to determine the deltas.
There is already a work around for this in place - if we detect a UID
being returned from one of these VANISHED responses that is older than
the oldest known UID we know was sent to the device then we reset the
device's mailbox state, causing the mailbox to resync.
We already track the UIDs of all email sent to the device in our state
tables when using a server that does not support QRESYNC, I plan on
extending/improving this and getting rid of the current use VANISHED
data completely.
For further information about our code vs ZPush you can read a basic
overview on our wiki page http://wiki.horde.org/ActiveSync. I'm in the
process of a more detailed feature comparison grid, but it's far from
ready.
Please move any further discussion specific to Horde's ActiveSync code
to the dev@lists.horde.org mailing list.
Hacking dovecot might be complex, as it should fit in so many stuff and setups
I would never suggest people should hack server code to get other code
to work :)
Anyway i agree many people using active sync in their smartphone these days and new outlook will also have a active sync mode, so there shouldnt stay any fixable problems ( where ever they may be ), but in my understanding the imap server ( dovecot ) acts more like a backhand here did you test against recent versions of cyrus etc, how is their behave in this case?
I have not personally tested against cyrus, but I know some of our
other devs use cyrus with the ActiveSync code base.
mike
The Horde Project (www.horde.org) mrubinsk@horde.org
Quoting Timo Sirainen tss@iki.fi:
On 6.11.2012, at 3.49, Michael J Rubinsky wrote:
That would require infinitely storing the modseq of when each message was expunged. Not very nice. Also the RFC talks a lot about this situation. The SELECT command has two optional parameters to optimize it.
The RFC *does* indicate that a server implementation could,
strictly speaking, be considered in compliance without remembering
modsequences for all expunged messages, but it does explicitly
discourage such implementations. From RFC 5162 [4.1]:Strictly speaking, a server implementation that doesn't remember mod- sequences associated with expunged messages can be considered compliant with this specification. Such implementations return all expunged messages specified in the UID set of the UID FETCH (VANISHED) command every time, without paying attention to the specified CHANGEDSINCE mod-sequence. Such implementations are discouraged, as they can end up returning VANISHED responses that are bigger than the result of a UID SEARCH command for the same UID set.
This is talking about a server that doesn't permanently remember ANY
modseqs for expunges. Dovecot remembers them, not not infinitely.It also gives advice to avoid infinitely storing the modsequences
such as "expiring" sequences associated with older expunged
messages, but assigning a single modsequence value to all of the
expired expunged messages.Dovecot behaves as the section 4.3 describes. Note especially:
Note that indefinitely storing information about expunged messages can cause storage and related problems for an implementation. .. Hence, implementations are encouraged to adopt strategies to protect against such storage problems, such as limiting the size of the queue used to store mod-sequences for expunged messages and "expiring" older records when this limit is reached. When the selected implementation-specific queue limit is reached, the oldest record(s) are deleted from the queue (note that such records are located at the queue head). For all such "expired" records, the server needs to store a single mod-sequence, which is the highest mod-sequence for all "expired" expunged messages.
This is exactly what Dovecot does. There is a single modseq
associated with all the previously expunged messages. If you try to
request expunges for that modseq, it returns all of the expunged
messages, which is what you're seeing as a problem.It might be helpful in some situations to decrease the precision and
remember for example:
- UIDs 1-100 were deleted with modseq 10 (in reality multiple times
between modseqs 1..10)- UIDs 101-130 were deleted with modseq 15 (in reality with modseqs 11..15) .. and so on
But this assumes that the expunged UID ranges compress well. If UIDs
are being deleted here and there it's still pretty wasteful to store
them.And yes, the current way may be forgetting them a bit too early when
a lot of other unrelated changes are happening. It would be possible
to keep a separate expunge log which could remember the expunges
longer. But that would be yet another different index file for
Dovecot, which annoyingly complicates everything. And currently
since it sounds like the only problem is activesync implementation
using it, I'm not very interested in spending a lot of time on it.
These defines in mail-transaction-log-private.h anyway can be
changed to make it much less likely to see your problem:/* Rotate when log is older than ROTATE_TIME and larger than MIN_SIZE */ #define MAIL_TRANSACTION_LOG_ROTATE_MIN_SIZE (1024*32) /* If log is larger than MAX_SIZE, rotate regardless of the time */ #define MAIL_TRANSACTION_LOG_ROTATE_MAX_SIZE (1024*1024) #define MAIL_TRANSACTION_LOG_ROTATE_TIME (60*5)
/* Delete .log.2 files older than this many seconds. Don't be too eager, older files are useful for QRESYNC and dsync. */ #define MAIL_TRANSACTION_LOG2_STALE_SECS (60*60*24*2)
Maybe the defaults could be changed..
Thanks for the information and clarification, Timo.
-- mike
The Horde Project (www.horde.org) mrubinsk@horde.org
On 6.11.2012, at 14.17, Timo Sirainen wrote:
It would be possible to keep a separate expunge log which could remember the expunges longer. But that would be yet another different index file for Dovecot, which annoyingly complicates everything. And currently since it sounds like the only problem is activesync implementation using it, I'm not very interested in spending a lot of time on it.
Here's an idea how it could maybe be done without too much complexity:
Before deleting/replacing dovecot.index.log.2 file, scan all of the expunges from it and append them to dovecot.index.log.expunges file using the normal transaction log file format. Change the reader code to use it only when a special flag is enabled (because it has no other changes, it shouldn't normally be used). Use that flag when looking for old expunges. Once in a while recreate the file and drop some of the oldest expunges.
The annoying thing with that is that the file format wastes disk space. An alternative would be to write to the file using yet another format optimized for it, and then have a separate "expunge scanner" API that scans the expunge file and the transaction logs.
Anyway, not something I have time to implement myself anytime soon.
Quoting Timo Sirainen tss@iki.fi:
On 6.11.2012, at 3.49, Michael J Rubinsky wrote:
These defines in mail-transaction-log-private.h anyway can be
changed to make it much less likely to see your problem:/* Rotate when log is older than ROTATE_TIME and larger than MIN_SIZE */ #define MAIL_TRANSACTION_LOG_ROTATE_MIN_SIZE (1024*32) /* If log is larger than MAX_SIZE, rotate regardless of the time */ #define MAIL_TRANSACTION_LOG_ROTATE_MAX_SIZE (1024*1024) #define MAIL_TRANSACTION_LOG_ROTATE_TIME (60*5)
/* Delete .log.2 files older than this many seconds. Don't be too eager, older files are useful for QRESYNC and dsync. */ #define MAIL_TRANSACTION_LOG2_STALE_SECS (60*60*24*2)
Maybe the defaults could be changed..
I'm not sure changing the defaults is a good idea. But if someone
does want to use a particular dovecot server as the backend for
activesync clients, for example, it would probably make sense to allow
these values to be tweaked via the config files. (I can see an
organization having a "normal" IMAP server and a "activesync" IMAP
server that differ in these details, and also in things like IDLE
timeouts).
michael
On 8.11.2012, at 0.08, Michael M Slusarz wrote:
These defines in mail-transaction-log-private.h anyway can be changed to make it much less likely to see your problem:
/* Rotate when log is older than ROTATE_TIME and larger than MIN_SIZE */ #define MAIL_TRANSACTION_LOG_ROTATE_MIN_SIZE (1024*32) /* If log is larger than MAX_SIZE, rotate regardless of the time */ #define MAIL_TRANSACTION_LOG_ROTATE_MAX_SIZE (1024*1024) #define MAIL_TRANSACTION_LOG_ROTATE_TIME (60*5)
/* Delete .log.2 files older than this many seconds. Don't be too eager, older files are useful for QRESYNC and dsync. */ #define MAIL_TRANSACTION_LOG2_STALE_SECS (60*60*24*2)
Maybe the defaults could be changed..
I'm not sure changing the defaults is a good idea. But if someone does want to use a particular dovecot server as the backend for activesync clients, for example, it would probably make sense to allow these values to be tweaked via the config files. (I can see an organization having a "normal" IMAP server and a "activesync" IMAP server that differ in these details, and also in things like IDLE timeouts).
Well .. I hate adding more settings. :) There are way too many already. Ideally Dovecot would automatically do the right thing anyway. Just like it already caches only those things that are needed. It could also increase these values when QRESYNC is used, or even better to actually have the separate expunge log that I mentioned.
Quoting Timo Sirainen tss@iki.fi:
On 8.11.2012, at 0.08, Michael M Slusarz wrote:
I'm not sure changing the defaults is a good idea. But if someone
does want to use a particular dovecot server as the backend for
activesync clients, for example, it would probably make sense to
allow these values to be tweaked via the config files. (I can see
an organization having a "normal" IMAP server and a "activesync"
IMAP server that differ in these details, and also in things like
IDLE timeouts).Well .. I hate adding more settings. :) There are way too many
already. Ideally Dovecot would automatically do the right thing
anyway. Just like it already caches only those things that are
needed. It could also increase these values when QRESYNC is used, or
even better to actually have the separate expunge log that I
mentioned.
Thinking about this more, this can really all be handled by proper MUA
design. In short, it is never a good idea to send a '1:*' UID range
to a VANISHED CHANGEDSINCE FETCH.
It remains a reasonable MUA design decision to not send the actual
cached UID list to the FETCH command: if this cached UID list is
thousands of messages long, obtaining this list, (optionally) sequence
set compressing, and sending via the command may take more
time/resources than it saves. But a MUA should, at a minimum, keep
track of the minimum UID it is aware of in order to limit the possible
response. This is a trivial amount of extra overhead and would
prevent a large number of spurious VANISHED UIDs to need to be
traversed.
michael
Looks like there was a bug in here: http://hg.dovecot.org/dovecot-2.2/rev/98195220a0f7
On 8.11.2012, at 0.08, Michael M Slusarz slusarz@curecanti.org wrote:
Quoting Timo Sirainen tss@iki.fi:
On 6.11.2012, at 3.49, Michael J Rubinsky wrote:
These defines in mail-transaction-log-private.h anyway can be changed to make it much less likely to see your problem:
/* Rotate when log is older than ROTATE_TIME and larger than MIN_SIZE */ #define MAIL_TRANSACTION_LOG_ROTATE_MIN_SIZE (1024*32) /* If log is larger than MAX_SIZE, rotate regardless of the time */ #define MAIL_TRANSACTION_LOG_ROTATE_MAX_SIZE (1024*1024) #define MAIL_TRANSACTION_LOG_ROTATE_TIME (60*5)
/* Delete .log.2 files older than this many seconds. Don't be too eager, older files are useful for QRESYNC and dsync. */ #define MAIL_TRANSACTION_LOG2_STALE_SECS (60*60*24*2)
Maybe the defaults could be changed..
I'm not sure changing the defaults is a good idea. But if someone does want to use a particular dovecot server as the backend for activesync clients, for example, it would probably make sense to allow these values to be tweaked via the config files. (I can see an organization having a "normal" IMAP server and a "activesync" IMAP server that differ in these details, and also in things like IDLE timeouts).
michael
Quoting Timo Sirainen tss@iki.fi:
On 6.11.2012, at 3.49, Michael J Rubinsky wrote:
That would require infinitely storing the modseq of when each message was expunged. Not very nice. Also the RFC talks a lot about this situation. The SELECT command has two optional parameters to optimize it.
The RFC *does* indicate that a server implementation could,
strictly speaking, be considered in compliance without remembering
modsequences for all expunged messages, but it does explicitly
discourage such implementations. From RFC 5162 [4.1]:Strictly speaking, a server implementation that doesn't remember mod- sequences associated with expunged messages can be considered compliant with this specification. Such implementations return all expunged messages specified in the UID set of the UID FETCH (VANISHED) command every time, without paying attention to the specified CHANGEDSINCE mod-sequence. Such implementations are discouraged, as they can end up returning VANISHED responses that are bigger than the result of a UID SEARCH command for the same UID set.
This is talking about a server that doesn't permanently remember ANY
modseqs for expunges. Dovecot remembers them, not not infinitely.It also gives advice to avoid infinitely storing the modsequences
such as "expiring" sequences associated with older expunged
messages, but assigning a single modsequence value to all of the
expired expunged messages.Dovecot behaves as the section 4.3 describes. Note especially:
Note that indefinitely storing information about expunged messages can cause storage and related problems for an implementation. .. Hence, implementations are encouraged to adopt strategies to protect against such storage problems, such as limiting the size of the queue used to store mod-sequences for expunged messages and "expiring" older records when this limit is reached. When the selected implementation-specific queue limit is reached, the oldest record(s) are deleted from the queue (note that such records are located at the queue head). For all such "expired" records, the server needs to store a single mod-sequence, which is the highest mod-sequence for all "expired" expunged messages.
This is exactly what Dovecot does. There is a single modseq
associated with all the previously expunged messages. If you try to
request expunges for that modseq, it returns all of the expunged
messages, which is what you're seeing as a problem.
I see your point, but the problem is that is not intuitive when
reading the RFC. One part of the RFC defines the behavior of VANISHED
(EARLIER) as only returning changes since the mod-sequence given. And
you are correct that another part of the RFC says that, essentially, a
server is allowed to break this required response.
I'm thinking that this is more of an issue with the way the RFC is
written. I'll move this over to the imap protocol list to get further
input.
michael
Quoting Michael M Slusarz slusarz@curecanti.org:
I see your point, but the problem is that is not intuitive when
reading the RFC. One part of the RFC defines the behavior of
VANISHED (EARLIER) as only returning changes since the mod-sequence
given. And you are correct that another part of the RFC says that,
essentially, a server is allowed to break this required response.I'm thinking that this is more of an issue with the way the RFC is
written. I'll move this over to the imap protocol list to get
further input.
Sigh. Never mind. For some reason, I completely ignored (missed?)
this part of the RFC:
Note: A server that receives a mod-sequence smaller than <minmodseq>,
where <minmodseq> is the value of the smallest expunged mod-sequence
it remembers minus one, MUST behave as if it was requested to report
all expunged messages from the provided UID set parameter.So you are right, I was wrong, and the world is good.
michael
On 8.11.2012, at 0.34, Michael M Slusarz wrote:
Quoting Michael M Slusarz slusarz@curecanti.org:
I see your point, but the problem is that is not intuitive when reading the RFC. One part of the RFC defines the behavior of VANISHED (EARLIER) as only returning changes since the mod-sequence given. And you are correct that another part of the RFC says that, essentially, a server is allowed to break this required response.
I'm thinking that this is more of an issue with the way the RFC is written. I'll move this over to the imap protocol list to get further input.
Sigh. Never mind. For some reason, I completely ignored (missed?) this part of the RFC:
Note: A server that receives a mod-sequence smaller than <minmodseq>, where <minmodseq> is the value of the smallest expunged mod-sequence it remembers minus one, MUST behave as if it was requested to report all expunged messages from the provided UID set parameter.
So you are right, I was wrong, and the world is good.
I wonder how much would it help if you
a) Used the uidset/seqset parameters with SELECT command
and optionally
b) Dovecot implemented it slightly better than required by RFC: http://www.ietf.org/mail-archive/web/lemonade/current/msg04771.html
Quoting Timo Sirainen tss@iki.fi:
On 8.11.2012, at 0.34, Michael M Slusarz wrote:
Quoting Michael M Slusarz slusarz@curecanti.org:
I see your point, but the problem is that is not intuitive when
reading the RFC. One part of the RFC defines the behavior of
VANISHED (EARLIER) as only returning changes since the
mod-sequence given. And you are correct that another part of the
RFC says that, essentially, a server is allowed to break this
required response.I'm thinking that this is more of an issue with the way the RFC is
written. I'll move this over to the imap protocol list to get
further input.Sigh. Never mind. For some reason, I completely ignored (missed?)
this part of the RFC:Note: A server that receives a mod-sequence smaller than <minmodseq>, where <minmodseq> is the value of the smallest expunged mod-sequence it remembers minus one, MUST behave as if it was requested to report all expunged messages from the provided UID set parameter.
So you are right, I was wrong, and the world is good.
I wonder how much would it help if you
a) Used the uidset/seqset parameters with SELECT command
We *do* use this information. However, this is not (necessarily)
useful for the activesync query that was the genesis of this thread. A
bit of background on our MUA design is necessary.
For Horde/IMP, all IMAP server configuration is done through the IMP
application. As part of this configuration, a cache backend can be
configured.
There are multiple potential users of this IMAP object. Within IMP
itself, multiple sessions can be open at any one time. Additionally,
several views of IMP, our dynamic view and our smartmobile view, have
another cache of messages kept on the browser side. Finally, the
ActiveSync library also uses the IMAP object configured by IMP.
Anytime the IMAP object is accessed, we are syncing the mailbox with
the IMP-configured cache. For QRESYNC, we use the SELECT/EXAMINE
uidset parameter.
The problem is that any particular view may not be sync'd to the same
state as the IMP cache. For example, if someone is using the web
application and their phone is syncing via ActiveSync, it is quite
likely that the activesync cached mod-sequence value will NOT equal
the IMP cached mod-sequence value. So this is when explicitly calling
FETCH VANISHED CHANGEDSINCE is needed.
The good news: once we get the CHANGEDSINCE FETCH information, we
don't need to do a separate flags sync since this information has
already been cached within the IMAP object (via either the
CHANGEDSINCE call or, more likely, a previous FETCH call in another
session). Further optimization: in the case where the original
QRESYNC/CONDSTORE sync matches the mod-sequence of whatever
object/view is accessing the IMAP object, which should be the most
common occurrence, there is no need to perform any additional
FETCH/SEARCH calls since we cache the results of the initial mailbox
sync and return this data.
Might be a long-winded explanation, but just wanted to show why FETCH
VANISHED CHANGEDSINCE MUST be used by a client even if taking
advantage of QRESYNC SELECT/EXAMINE syncing. In other words - I'd
like to think that my imap implementation is not broken :)
b) Dovecot implemented it slightly better than required by RFC: http://www.ietf.org/mail-archive/web/lemonade/current/msg04771.html
I spent a week or so trying to cache message sequence number -> UID
mapping. And determined it was more trouble than it was worth.
The gains from more compact VANISHED responses in SELECT/EXAMINE are
minimal compared to the expense to track them. And the only other
reason for tracking - the possibility that EXPUNGEs return EXPUNGED
responses instead of VANISHED if the UIDs of the actually expunged
messages are needed - can be worked around by doing a UID SEARCH call
after the EXPUNGE is over and comparing to the list of UIDs that were
given to UID EXPUNGE (with the further optimization that I cache
MSN->UID while in a mailbox, which should catch the "STORE
(\Deleted)/EXPUNGE" common when using a Trash mailbox or immediate
message deletion). Unless I am missing something else that MSNs are
necessary?
michael
participants (4)
-
Michael J Rubinsky
-
Michael M Slusarz
-
Robert Schetterer
-
Timo Sirainen