[Dovecot] Bug in HIGHESTMODSEQ when MODSEQ tracking is not enabled for version 2.0.18.
Greetings,
I'm new to the list. I have encountered what I believe to be a bug in the return of HIGHESTMODSEQ when talking to a IMAP client. HIGHESTMODSEQ does not appear to be incrementing when a new message arrives. The exact steps I took are:
- Create a new mail account
- Send a mail to the account, check that you can receive the email. Send a second mail to the account.
- perform a SELECT on the mailbox from a new session.
PMLK4 SELECT INBOX
- FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
- OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft \*)] Flags permitted.
- 2 EXISTS
- 1 RECENT
- OK [UNSEEN 1] First unseen.
- OK [UIDVALIDITY 1334074920] UIDs valid
- OK [UIDNEXT 3] Predicted next UID
- OK [HIGHESTMODSEQ 1] Highest PMLK4 OK [READ-WRITE] Select completed.
- Retrieve the mail. Then perform another SELECT on the mailbox from yet another session.
LLEP4 SELECT INBOX
- FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
- OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft \*)] Flags permitted.
- 2 EXISTS
- 0 RECENT
- OK [UNSEEN 1] First unseen.
- OK [UIDVALIDITY 1334074920] UIDs valid
- OK [UIDNEXT 3] Predicted next UID
- OK [HIGHESTMODSEQ 1] Highest LLEP4 OK [READ-WRITE] Select completed.
- Send another mail to the account, and check SELECT again.
FKDA4 SELECT INBOX
- FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
- OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft \*)] Flags permitted.
- 3 EXISTS
- 1 RECENT
- OK [UNSEEN 1] First unseen.
- OK [UIDVALIDITY 1334074920] UIDs valid
- OK [UIDNEXT 4] Predicted next UID
- OK [HIGHESTMODSEQ 1] Highest FKDA4 OK [READ-WRITE] Select completed.
It appears as if HIGHESTMODSEQ is not being updated. I can get HIGHESTMODSEQ to start updating correctly if I send a "UID FETCH 1 MODSEQ" or similar command, which appears to enable MODSEQ tracking at the server (according to the comment around line 173 in file src/lib-index/mail-index-modseq.c), but until that command is sent, MODSEQ tracking is not enabled and hence HIGHESTMODSEQ is always going to return 1.
According to RFC4551, the combination of HIGHESTMODSEQ and UIDVALIDITY should be sufficient to determine if the metadata associated with the mailbox has changed, but in this case looking at only those two parameters does not yield sufficient information about changes in the mailbox. The mail client I'm using relies on the combination of HIGHESTMODSEQ and UIDVALIDITY to determine if there are changes in the mailbox, and hence does not see new mail come in.
It seems that Dovecot should not be returning HIGHESTMODSEQ in response to a command if MODSEQ tracking is not enabled, but I could be wrong. I've attached my configuration (it's Dovecot 2.0.18 running on Debian Stable).
Let me know if you need more information.
Glenn.
This transmission (including any attachments) may contain confidential information, privileged material (including material protected by the solicitor-client or other applicable privileges), or constitute non-public information. Any use of this information by anyone other than the intended recipient is prohibited. If you have received this transmission in error, please immediately reply to the sender and delete this information from your system. Use, dissemination, distribution, or reproduction of this transmission by unintended recipients is not authorized and may be unlawful.
On 10.4.2012, at 20.17, Glenn Wurster wrote:
It appears as if HIGHESTMODSEQ is not being updated. I can get HIGHESTMODSEQ to start updating correctly if I send a "UID FETCH 1 MODSEQ" or similar command, which appears to enable MODSEQ tracking at the server (according to the comment around line 173 in file src/lib-index/mail-index-modseq.c), but until that command is sent, MODSEQ tracking is not enabled and hence HIGHESTMODSEQ is always going to return 1.
Yes, modseqs aren't tracked in a mailbox until client expresses an interest for them. It would be a waste of disk space to save them since 99% of users don't need them.
According to RFC4551, the combination of HIGHESTMODSEQ and UIDVALIDITY should be sufficient to determine if the metadata associated with the mailbox has changed, but in this case looking at only those two parameters does not yield sufficient information about changes in the mailbox. The mail client I'm using relies on the combination of HIGHESTMODSEQ and UIDVALIDITY to determine if there are changes in the mailbox, and hence does not see new mail come in.
Yeah, it does seem that the RFC says that..
It seems that Dovecot should not be returning HIGHESTMODSEQ in response to a command if MODSEQ tracking is not enabled, but I could be wrong. I've attached my configuration (it's Dovecot 2.0.18 running on Debian Stable).
RFC 4551 says that HIGHESTMODSEQ or NOMODSEQ MUST be returned. Hmm. Perhaps:
If the session is known to have modseqs enabled, immediately enable modseqs for newly created mailboxes
If a mailbox doesn't have modseqs enabled, return NOMODSEQ. This isn't ideal, but seems like the only possibility.
Yes, modseqs aren't tracked in a mailbox until client expresses an interest for them. It would be a waste of disk space to save them since 99% of users don't need them.
Makes sense, our mail client gets caught in the middle though, because it uses HIGHESTMODSEQ to track mailbox updates without using MODSEQ options on SELECT/FETCH to track message updates.
RFC 4551 says that HIGHESTMODSEQ or NOMODSEQ MUST be returned. Hmm. Perhaps:
- If the session is known to have modseqs enabled, immediately enable modseqs for newly created mailboxes
MODSEQ seems to be enabled on the mailbox, not on the session. For disconnected IMAP, there will frequently be new sessions established, and so we may not have a good indicator of whether the client is using MODSEQ (based on my reading). The current approach of looking for a MODSEQ related command seems to work in most cases, and so I'd be tempted to keep the current functionality (minus the incorrect reporting of HIGHESTMODSEQ if we can find a solution).
- If a mailbox doesn't have modseqs enabled, return NOMODSEQ. This isn't ideal, but seems like the only possibility.
The RFC also states that if we return NOMODSEQ we'd have to return a tagged BAD response to "UID FETCH 1 MODSEQ", which appears to one of the commands that enables MODSEQ for Dovecot ("SELECT INBOX (CONDSTORE)" also enables it...). What about returning a BAD response and at the same time start tracking MODSEQ so that future SELECT commands would return HIGHESTMODSEQ? Do we know what email clients are using CONDSTORE options and how they'd react to a mailbox suddenly having MODSEQ capabilities after we just told them it didn't?
Glenn.
This transmission (including any attachments) may contain confidential information, privileged material (including material protected by the solicitor-client or other applicable privileges), or constitute non-public information. Any use of this information by anyone other than the intended recipient is prohibited. If you have received this transmission in error, please immediately reply to the sender and delete this information from your system. Use, dissemination, distribution, or reproduction of this transmission by unintended recipients is not authorized and may be unlawful.
On 10.4.2012, at 23.50, Glenn Wurster wrote:
Yes, modseqs aren't tracked in a mailbox until client expresses an interest for them. It would be a waste of disk space to save them since 99% of users don't need them.
Makes sense, our mail client gets caught in the middle though, because it uses HIGHESTMODSEQ to track mailbox updates without using MODSEQ options on SELECT/FETCH to track message updates.
It would be actually possible for Dovecot to always keep track of highestmodseq, even if individual modseqs weren't tracked. I almost implemented it, but keeping it backwards compatible with old versions would have needed to make it more complex. Maybe v2.2 could do this.
- If a mailbox doesn't have modseqs enabled, return NOMODSEQ. This isn't ideal, but seems like the only possibility.
The RFC also states that if we return NOMODSEQ we'd have to return a tagged BAD response to "UID FETCH 1 MODSEQ", which appears to one of the commands that enables MODSEQ for Dovecot ("SELECT INBOX (CONDSTORE)" also enables it...). What about returning a BAD response and at the same time start tracking MODSEQ so that future SELECT commands would return HIGHESTMODSEQ? Do we know what email clients are using CONDSTORE options and how they'd react to a mailbox suddenly having MODSEQ capabilities after we just told them it didn't?
That's kind of an annoying part of the RFC that it says the commands MUST fail with BAD.. I don't think there was really any good reason to add that text. Also Dovecot hasn't failed those commands earlier also with mailbox formats that don't support modseqs at all. So at least for now I simply made it return NOMODSEQ when modseqs aren't enabled, and the rest of the behavior is the same.
participants (2)
-
Glenn Wurster
-
Timo Sirainen