Fwd: HIGHESTMODSEQ tracking
Michael,
2014-07-10 1:37 GMT+04:00 Michael M Slusarz slusarz@curecanti.org:
Quoting Kostya Vasilyev kmansoft@gmail.com:
Hello,
I'd like to follow up on someone else's old thread:
http://dovecot.org/list/dovecot/2012-April/082624.html
I understand that Dovecot (that 2012 discussion was about version 2.0.18) can return "HIGHESTMODSEQ 1" upon SELECT if persistent modseq tracking was never enabled for a mailbox.
This behavior is largely irrelevant ... a client should ignore any HIGHESTMODSEQ data that is sent if it (the client) hasn't sent a CONDSTORE enabling command.
True, but I was just pointing out a quirk that existed in a older version, to ask for a clarification.
I'd like to get a clarification on how to enable proper persistent modseq
tracking.
Per RFC 4551, "A client supporting CONDSTORE.... indicates its willingness to receive mod-sequence updates... by issuing:"
SELECT mailbox (CONDSTORE) STATUS (... HIGHESTMODSEQ ...) ... others ...
4551 is deprecated --> it is now RFC 7162.
Thanks, didn't notice that 4551 was "obsoleted".
Also note there's actually 6 different categories of CONDSTORE-enabling commands. SELECT/EXAMINE with CONDSTORE is just one of the categories.
There are five listed in RFC 4551, yes, there are six in 7162, but my question was about two specific ones out of those five / six.
These days, you *really* should be using QRESYNC instead though.
There are some mail servers that support CONDSTORE but not QRESYNC. The old chicken and egg IMAP problem :)
It seems that Dovecot 2.1.7 (Debian 7.5) enables persistent modseq
tracking after one of the above two commands (and maybe others). So far so good.
Incorrect. Dovecot has supported CONDSTORE/QRESYNC since 1.2.0.
Incorrect?
I wrote: "it seems that Dovecot 2.1.7 enables persistent modseq tracking...".
Are you implying that Dovecot 2.1.7 doesn't enable persistent modseq tracking in this case?
I never claimed that earlier versions of Dovecot didn't advertise CONDSTORE or didn't implement it properly, etc.
Perhaps I should have written "...after *any* of the two commands... (and maybe others)" -- meaning it does after either of those two commands, and the other ones I don't care about and so didn't test.
And as mentioned above, there are 6 different ways of enabling CONDSTORE. (Enabling QRESYNC is done 1 way ... the "ENABLE QRESYNC" command).
Yes, agreed, noted.
I'm asking about two out of those five / six.
And not just about what the RFC says, but about how real, out in the field, installed on mail servers that people use, possibly quite old, versions of Dovecot behave.
So let me try again:
Is sending SELECT ... (CONDSTORE) or STATUS (... HIGHESTMODSEQ ...) enough to enable reliable persistent modseq tracking for all versions of Dovecot which advertise CONDSTORE in their capabilities?
Not what the currently relevant RFC says, but how Dovecot versions with CONDSTORE in their caps behave.
This is the one thing I'd really like to know.
So my questions are:
1 - Is this (enabling persistent modseq) also the case for all Dovecot versions prior to 2.1.7 that advertise CONDSTORE?
There must be people running older versions out there, older than 2.1.7.
If not, how can I branch my logic to not use HIGHESTMODSEQ, given that Dovecot doesn't appear to return any version info via its greeting or via the ID command?
As noted above, this is irrelevant. As a client author, you should ALWAYS ignore HIGHESTMODSEQ if you haven't enabled CONDSTORE/QRESYNC .
All caps look nice in RFCs, but the real world has more variety :)
3 - I guess the change to return NOMOSEQ instead of "1" for mailboxes with
modseq tracking disabled, as mentioned in that discussion from 2012, never happened? At least on the 2.1 branch?
NOMODSEQ is only relevant *if* you have enabled CONDSTORE.
Agreed. However:
Returning NOMODSEQ on mailboxes that don't support persistent mod-sequences has worked since 1.2.0, at least as far as I know.
The linked message from 2012, discussing 2.0.18, says:
"2) If a mailbox doesn't have modseqs enabled, return NOMODSEQ. This isn't ideal, but seems like the only possibility"
And based on my tests, Dovecot 2.1.7 returns "HIGHESTMODSEQ 1" for mailboxes where modseq tracking has not been enabled.
You're largely right about the value only being relevant if the client enabled CONDSTORE, and I don't care too much about this quirk, just wanted to to get a clarification on this as well, if possible.
Thanks, -- K
Quoting Kostya Vasilyev kmansoft@gmail.com:
2014-07-10 1:37 GMT+04:00 Michael M Slusarz slusarz@curecanti.org:
Quoting Kostya Vasilyev kmansoft@gmail.com:
These days, you *really* should be using QRESYNC instead though.
There are some mail servers that support CONDSTORE but not QRESYNC. The old chicken and egg IMAP problem :)
This is the wrong way to look at it though.
You should implement QRESYNC initially, because it will force you to
implement CONDSTORE also.
It's going to be 10x harder to implement CONDSTORE and then add
QRESYNC functionality on top of it later. Do it right the first time.
Both Dovecot and Cyrus support both CONDSTORE and QRESYNC, and
combined that is more than 50% market share, so that should be
incentive enough. Gmail only supports CONDSTORE, but it's the outlier.
So let me try again:
Is sending SELECT ... (CONDSTORE) or STATUS (... HIGHESTMODSEQ ...) enough to enable reliable persistent modseq tracking for all versions of Dovecot which advertise CONDSTORE in their capabilities?
Maybe. You can't tell until you actually see whether the
EXAMINE/SELECT returns HIGHESTMODSEQ or NOMODSEQ.
3 - I guess the change to return NOMOSEQ instead of "1" for mailboxes with
modseq tracking disabled, as mentioned in that discussion from 2012, never happened? At least on the 2.1 branch?
NOMODSEQ is only relevant *if* you have enabled CONDSTORE.
Agreed. However:
Returning NOMODSEQ on mailboxes that don't support persistent mod-sequences has worked since 1.2.0, at least as far as I know.
The linked message from 2012, discussing 2.0.18, says:
"2) If a mailbox doesn't have modseqs enabled, return NOMODSEQ. This isn't ideal, but seems like the only possibility"
AFAICT, the mailing list message you are referring to is talking about
what Dovecot returns **when CONDSTORE is not enabled**. As I
mentioned before, this should be irrelevant to your client since you
shouldn't be using HIGHESTMODSEQ if you haven't enabled CONDSTORE.
Older versions of Dovecot happen to send CONDSTORE info even if it's
not active (which is perfectly valid IMAP behavior). *This* was the
issue referred to in that message ... some clients didn't enable
CONDSTORE but were using HIGHESTMODSEQ if it existed to use it as a
"Uniqu eID" (combined with the current UIDVALIDITY) of the mailbox for
polling purposes. It was these clients that were seeing broken
behavior since this "Unique ID" never changed - since UIDVALIDITY
normally won't ever change - since HIGHESTMODSEQ would always be 1.
These clients should have never been doing that in the first place,
but whatever.
Once you issue a CONDSTORE-enabling command, this is no longer an
issue. So it's not something a client ever has to workaround as long
as they are following the RFC.
michael
2014-07-10 4:05 GMT+04:00 Michael M Slusarz slusarz@curecanti.org:
Quoting Kostya Vasilyev kmansoft@gmail.com:
2014-07-10 1:37 GMT+04:00 Michael M Slusarz slusarz@curecanti.org:
Quoting Kostya Vasilyev kmansoft@gmail.com:
These days, you *really* should be using QRESYNC instead though.
There are some mail servers that support CONDSTORE but not QRESYNC. The old chicken and egg IMAP problem :)
[ snip ] Both Dovecot and Cyrus support both CONDSTORE and QRESYNC, and combined that is more than 50% market share, so that should be incentive enough. Gmail only supports CONDSTORE, but it's the outlier.
Gmail still does have a few users, though. A few dozen at least, maybe more :)
And it has a big advantage, from my point of view, over Cyrus / Dovecot -- there is but one server version that's consistent for all accounts.
Yes, they do some things wrong (like not sending message flags changes over IDLE connections), but I can test something in my personal account, get feedback from 3-5-10 users with @gmail accounts, and be reasonably confident that everything is fine (and that I'd know know if it's not).
Now, Cyrus and Dovecot (and Courier I guess) is a different story, there is a variety of versions out there, and software being software, there may be bugs / glitches / quirks.
Since you mention Cyrus, do you know that it often (like, almost always) responds to "ID" with a "NO"? This is not RFC compliant, but it's what is actually does.
For the "more than 50% market share" of Dovecot / Cyrus, do you have a breakdown by version number? At least in terms of 1.* vs 2.0 and higher?
So let me try again:
Is sending SELECT ... (CONDSTORE) or STATUS (... HIGHESTMODSEQ ...) enough to enable reliable persistent modseq tracking for all versions of Dovecot which advertise CONDSTORE in their capabilities?
Maybe. You can't tell until you actually see whether the EXAMINE/SELECT returns HIGHESTMODSEQ or NOMODSEQ.
Are you saying that Dovecot will always (*will always*, and I mean *always*) return NOMODSEQ after a client "expresses interested in modseq values" and the server can't enable it for some reason?
Or if it was previously enabled, and then well, I don't know, "something happened"?
By *always* I mean -- since Dovecot first started having a CONDSTORE in its CAPS, including version a.b.c that came with now really old Debian X, and version h.j.k that came with now really old RHEL Y, but which are still out there on actual mail servers, being used in actual mail accounts?
When something goes wrong in an email app, then to the user, it's always the email app developer's fault. Nobody gives a damn about the subtleties of what RFC abc says about xyz, or if server version j.k.l from three years ago had a bug.
So, before enabling certain optimizations for Dovecot, I thought I'd ask on a Dovecot mailing list, about actual behavior for this server feature.
I assume this mailing list has people with real Dovecot experience and knowledge, and maybe even the developers are lurking here too.
Specifically, I was hoping to hear back maybe something like this:
"Dovecot version X which was packaged in Debian Z, would not update the modseq value after command Y".
Or maybe -- which would be great:
"There were no issues with modseq tracking, at all, no reported bugs, code not touched, since the feature was originally implemented and advertised as CONDSTORE in CAPS in version 1.2.*".
-- K
Quoting Kostya Vasilyev kmansoft@gmail.com:
2014-07-10 4:05 GMT+04:00 Michael M Slusarz slusarz@curecanti.org:
Quoting Kostya Vasilyev kmansoft@gmail.com:
2014-07-10 1:37 GMT+04:00 Michael M Slusarz slusarz@curecanti.org:
Quoting Kostya Vasilyev kmansoft@gmail.com:
These days, you *really* should be using QRESYNC instead though.
There are some mail servers that support CONDSTORE but not QRESYNC. The old chicken and egg IMAP problem :)
[ snip ] Both Dovecot and Cyrus support both CONDSTORE and QRESYNC, and combined that is more than 50% market share, so that should be incentive enough. Gmail only supports CONDSTORE, but it's the outlier.
Gmail still does have a few users, though. A few dozen at least, maybe more :)
And it has a big advantage, from my point of view, over Cyrus / Dovecot -- there is but one server version that's consistent for all accounts.
Yes, they do some things wrong (like not sending message flags changes over IDLE connections), but I can test something in my personal account, get feedback from 3-5-10 users with @gmail accounts, and be reasonably confident that everything is fine (and that I'd know know if it's not).
This is getting a bit off-topic on this list... but Gmail does a LOT
of things wrong. Head over to one of the IMAP lists for further
information.
If you are testing against Gmail as the gold standard as to how a IMAP
server should operate, I can safely say you are Doing It Wrong.
For the "more than 50% market share" of Dovecot / Cyrus, do you have a breakdown by version number? At least in terms of 1.* vs 2.0 and higher?
I do not.
Maybe. You can't tell until you actually see whether the EXAMINE/SELECT returns HIGHESTMODSEQ or NOMODSEQ.
Are you saying that Dovecot will always (*will always*, and I mean *always*) return NOMODSEQ after a client "expresses interested in modseq values" and the server can't enable it for some reason?
Much like UIDVALIDITY should never change, NOMODSEQ will never be sent
(practical usage) for an active CONDSTORE access. You are asking
about a tremendously rare occurrence.
The whole deal with "HIGHESTMODSEQ 1" is irrelevant if you enable
CONDSTORE. I can't tell you what a server will return if you enable
CONDSTORE in one session, but then don't in another. But that doesn't
matter, since you aren't using HIGHESTMODSEQ in the latter case. As
long as CONDSTORE is active, HIGHESTMODSEQ will be updated, at least
in my 6 year experience with Dovecot which involves handling
installations with millions of users.
Or if it was previously enabled, and then well, I don't know, "something happened"?
By *always* I mean -- since Dovecot first started having a CONDSTORE in its CAPS, including version a.b.c that came with now really old Debian X, and version h.j.k that came with now really old RHEL Y, but which are still out there on actual mail servers, being used in actual mail accounts?
I have never run into an issue with HIGHESTMODSEQ for a properly
CONDSTORE-enabled session for Dovecot ever. I was one of the first
people (that I am aware of) that implemented CONDSTORE/QRESYNC back in
the early days (2009) ... and Dovecot was exclusively the server I was
developing with at that time.
When something goes wrong in an email app, then to the user, it's always the email app developer's fault. Nobody gives a damn about the subtleties of what RFC abc says about xyz, or if server version j.k.l from three years ago had a bug.
Agree, but only up to a certain point. If something is so onerous to
work around, then it *is* ok to say "it's the server's fault and we're
not going to work around this." Like everything else in life, there
is a cost/benefit analysis that must be done to determine where that
line needs to be drawn.
So, before enabling certain optimizations for Dovecot, I thought I'd ask on a Dovecot mailing list, about actual behavior for this server feature.
I assume this mailing list has people with real Dovecot experience and knowledge, and maybe even the developers are lurking here too.
Specifically, I was hoping to hear back maybe something like this:
"Dovecot version X which was packaged in Debian Z, would not update the modseq value after command Y".
Or maybe -- which would be great:
"There were no issues with modseq tracking, at all, no reported bugs, code not touched, since the feature was originally implemented and advertised as CONDSTORE in CAPS in version 1.2.*".
There are certainly bugs - I found several of them years ago when the
code was brand new (here's a thread:
http://markmail.org/message/fj74xta5z5uv4nix). But nothing that was
showstopping. And none of those versions are being run anymore for
all intents and purposes.
The bigger issue, specifically with QRESYNC, is not implementation
bugs but rather some deficiencies in the original standards (e.g.
unsolicited FETCHs without UIDs; VANISHED wasn't a required response
so sequence number tracking was still required) that weren't addressed
until RFC 7162. Those are more likely to trip you up then some
transient implementation bug.
michael
2014-07-15 3:47 GMT+04:00 Michael M Slusarz slusarz@curecanti.org:
Quoting Kostya Vasilyev kmansoft@gmail.com:
Gmail still does have a few users, though. A few dozen at least, maybe more :)
And it has a big advantage, from my point of view, over Cyrus / Dovecot -- there is but one server version that's consistent for all accounts.
Yes, they do some things wrong (like not sending message flags changes over IDLE connections), but I can test something in my personal account, get feedback from 3-5-10 users with @gmail accounts, and be reasonably confident that everything is fine (and that I'd know know if it's not).
This is getting a bit off-topic on this list... but Gmail does a LOT of things wrong. Head over to one of the IMAP lists for further information.
This is just one glaring example. Maybe you've ran into more than I have.
In any case, the point stands - with Gmail, it's much easier to be confident, from actual testing, that things works a certain way.
If you are testing against Gmail as the gold standard as to how a IMAP server should operate,
I never said or implied that. In fact, I pointed out a serious issue with Gmail's IMAP IDLE implementation, which means the exact opposite of holding it as a gold standard.
I can safely say you are Doing It Wrong.
It seems you enjoy pointing out to people when they're "wrong" or "incorrect" so much, you actually put meaning into their words that's not there? Or it it just me?
For the "more than 50% market share" of Dovecot / Cyrus, do you have a
breakdown by version number? At least in terms of 1.* vs 2.0 and higher?
I do not.
And without being able to get a version number from a Dovecot session (or so it seems to me -- nothing returned from ID...).... it looks kind of sad.
Maybe. You can't tell until you actually see whether the EXAMINE/SELECT
returns HIGHESTMODSEQ or NOMODSEQ.
Are you saying that Dovecot will always (*will always*, and I mean *always*) return NOMODSEQ after a client "expresses interested in modseq values" and the server can't enable it for some reason?
Much like UIDVALIDITY should never change, NOMODSEQ will never be sent (practical usage) for an active CONDSTORE access. You are asking about a tremendously rare occurrence.
In theory, yes, but I just wouldn't want to be surprised (and surprise my users).
The whole deal with "HIGHESTMODSEQ 1" is irrelevant if you enable CONDSTORE. I can't tell you what a server will return if you enable CONDSTORE in one session, but then don't in another. But that doesn't matter, since you aren't using HIGHESTMODSEQ in the latter case. As long as CONDSTORE is active, HIGHESTMODSEQ will be updated, at least in my 6 year experience with Dovecot which involves handling installations with millions of users.
Thank you.
This is the type of response, based on actual real-world experience, that I was looking for.
Or if it was previously enabled, and then well, I don't know, "something
happened"?
By *always* I mean -- since Dovecot first started having a CONDSTORE in its CAPS, including version a.b.c that came with now really old Debian X, and version h.j.k that came with now really old RHEL Y, but which are still out there on actual mail servers, being used in actual mail accounts?
I have never run into an issue with HIGHESTMODSEQ for a properly CONDSTORE-enabled session for Dovecot ever. I was one of the first people (that I am aware of) that implemented CONDSTORE/QRESYNC back in the early days (2009) ... and Dovecot was exclusively the server I was developing with at that time.
Great. Thank you again for a data point that comes from the real world.
When something goes wrong in an email app, then to the user, it's always
the email app developer's fault. Nobody gives a damn about the subtleties of what RFC abc says about xyz, or if server version j.k.l from three years ago had a bug.
Agree, but only up to a certain point. If something is so onerous to work around, then it *is* ok to say "it's the server's fault and we're not going to work around this." Like everything else in life, there is a cost/benefit analysis that must be done to determine where that line needs to be drawn.
Using modseq is an optimization. An optimization that makes things not work is not something I'd like to have.
So, before enabling certain optimizations for Dovecot, I thought I'd ask
on a Dovecot mailing list, about actual behavior for this server feature.
[snip]
There are certainly bugs - I found several of them years ago when the code was brand new (here's a thread: http://markmail.org/message/ fj74xta5z5uv4nix). But nothing that was showstopping. And none of those versions are being run anymore for all intents and purposes.
Thanks.
It's somewhat worrying that enabling CONDSTORE just once will cause the server to always track modseq values from that point on -- causing new code paths to execute.
But again, thanks for your data points rooted in the real world.
-- K
Quoting Kostya Vasilyev kmansoft@gmail.com:
This is getting a bit off-topic on this list... but Gmail does a LOT of things wrong. Head over to one of the IMAP lists for further information.
This is just one glaring example. Maybe you've ran into more than I have.
In any case, the point stands - with Gmail, it's much easier to be confident, from actual testing, that things works a certain way.
If you are testing against Gmail as the gold standard as to how a IMAP server should operate,
I never said or implied that. In fact, I pointed out a serious issue with Gmail's IMAP IDLE implementation, which means the exact opposite of holding it as a gold standard.
I can safely say you are Doing It Wrong.
It seems you enjoy pointing out to people when they're "wrong" or "incorrect" so much, you actually put meaning into their words that's not there? Or it it just me?
I was just trying to point out that this statement is very
dangerous/incorrect:
"In any case, the point stands - with Gmail, it's much easier to be
confident, from actual testing, that things works a certain way."
Gmail behavior may/can/will change overnight, and you will have no
idea. It makes a lot more sense to pick a local server of a known
version, that has deterministic behavior, to develop with.
For the "more than 50% market share" of Dovecot / Cyrus, do you have a
breakdown by version number? At least in terms of 1.* vs 2.0 and higher?
I do not.
And without being able to get a version number from a Dovecot session (or so it seems to me -- nothing returned from ID...).... it looks kind of sad.
ID extension is pretty much worthless for version identification. It
is trivially spoofed -- and some servers do exactly this in the real
world. All it takes is one server/version to be spoofed to make that
data worthless.
It's possible to do some level of basic version sniffing by things
like banner messages, Human-readable responses, CAPABILITY lists, and
ordering of responses to various commands. However, this is really
only useful for broad statistical surveys and not precise version
determination.
I have been able to work around all IMAP issues that have been
reported to us solely based on the data returned, rather than knowing
what IMAP server/version I am connected to.
michael
2014-07-16 0:13 GMT+04:00 Michael M Slusarz slusarz@curecanti.org:
Quoting Kostya Vasilyev kmansoft@gmail.com:
It seems you enjoy pointing out to people when they're "wrong" or "incorrect" so much, you actually put meaning into their words that's not there? Or it it just me?
I was just trying to point out that this statement is very dangerous/incorrect:
"In any case, the point stands - with Gmail, it's much easier to be confident, from actual testing, that things works a certain way."
Gmail behavior may/can/will change overnight, and you will have no idea.
Yes I will have a very good idea.
First, I use GMail myself, and second, I'll get messages from my users if the changes cause something in my app to break.
And because Gmail's software is pretty much identical for everyone (less the staged rollouts that I'm sure they use), I can see those changes very easily (one such case happened just recently).
And not just for Gmail -- I get messages from users about other issues too, and then I can react by:
putting a banner on my web site / forum (e.g. Yahoo recently letting their SSL certificate for SMTP expire)
or making changes to my software
It makes a lot more sense to pick a local server of a known version, that has deterministic behavior, to develop with.
For development, I use about a dozen test accounts, from Fastmail (probably my overall favorite) to Gmail, Yahoo, Dovecot, Yandex, Hotmail, and a few more that you've probably not heard of.
However, I'm not at liberty to pick mail servers / services for my users.
Sometimes the decision is made for them by someone else -- a corporate mail system, or a web hosting company.
Assuming latest versions of mail server software in those cases would be foolish on my part.
And yes, sometimes I look at the app's network logs and tell the user "you won't be able to do this, the server is broken".
However, I'd like to avoid such cases if I can, and sometimes have to implement compatibility hacks.
For the "more than 50% market share" of Dovecot / Cyrus, do you have a
breakdown by version number? At least in terms of 1.* vs 2.0 and higher?
I do not.
And without being able to get a version number from a Dovecot session (or so it seems to me -- nothing returned from ID...).... it looks kind of sad.
ID extension is pretty much worthless for version identification. It is trivially spoofed -- and some servers do exactly this in the real world. All it takes is one server/version to be spoofed to make that data worthless.
I'm not talking about doing it for security purposes, and for compatibility, spoofing seems to me like less of an issue.
In the realm of technically possible, someone could even write a random text generator and run it on port 993.
Why, GoDaddy has been doing exactly this for years, and they're even able to charge their customers for it.
It's possible to do some level of basic version sniffing by things like banner messages, Human-readable responses, CAPABILITY lists, and ordering of responses to various commands. However, this is really only useful for broad statistical surveys and not precise version determination.
Well, in theory, CAPABILITY is all you need, because all mail servers / services are strictly RFC compliant...
...and when they're not, they get fixed / upgraded very quickly...
...as soon as just one user relays a message from the developer of some "random" mail app to the mail service's support.
Hahaha.
I'd feel more confident about enabling CONDSTORE / modseq for Dovecot if I could exclude versions below 2.0, just to be safe. This has more practical value to me than a statistical survey.
Oh well.
Anyway, this thread has gotten quite far off-topic, thanks again for sharing your real-world experiences with Dovecot and its CONDSTORE / modseq support.
-- K
On 15 Jul 2014, at 23:13, Michael M Slusarz slusarz@curecanti.org wrote:
For the "more than 50% market share" of Dovecot / Cyrus, do you have a
breakdown by version number? At least in terms of 1.* vs 2.0 and higher?
I do not.
And without being able to get a version number from a Dovecot session (or so it seems to me -- nothing returned from ID...).... it looks kind of sad.
ID extension is pretty much worthless for version identification. It is trivially spoofed -- and some servers do exactly this in the real world. All it takes is one server/version to be spoofed to make that data worthless.
It's possible to do some level of basic version sniffing by things like banner messages, Human-readable responses, CAPABILITY lists, and ordering of responses to various commands. However, this is really only useful for broad statistical surveys and not precise version determination.
Here are some interesting results based on such survey: http://openemailsurvey.org/dovecot-versions.png
participants (3)
-
Kostya Vasilyev
-
Michael M Slusarz
-
Timo Sirainen