[Dovecot] automatic flag updates
Is there a way to get dovecot to send automatic flag updates? I notice that when I have two clients connected and one of them reads a new message, the server sends a status update which contains the number of unseen messages, but doesn't specifically say which ones have changed. I think the solution to this problem, while not wasting bandwidth, is to issue a unilateral FETCH FLAGS response whenever a flag changes. There doesn't seem to be a way to get dovecot (I'm using 0.99.10) to do this. Can the 1.0-test versions do this?
cya .sig
On 17.11.2004, at 02:34, Aaron VanDevender wrote:
Is there a way to get dovecot to send automatic flag updates? I notice that when I have two clients connected and one of them reads a new message, the server sends a status update which contains the number of unseen messages, but doesn't specifically say which ones have changed. I think the solution to this problem, while not wasting bandwidth, is to issue a unilateral FETCH FLAGS response whenever a flag changes. There doesn't seem to be a way to get dovecot (I'm using 0.99.10) to do this. Can the 1.0-test versions do this?
Dovecot has done this always. Most clients are just too stupid to do anything with them.
On Mon, 2004-11-29 at 05:36 +0200, Timo Sirainen wrote:
On 17.11.2004, at 02:34, Aaron VanDevender wrote:
Is there a way to get dovecot to send automatic flag updates? I notice that when I have two clients connected and one of them reads a new message, the server sends a status update which contains the number of unseen messages, but doesn't specifically say which ones have changed. I think the solution to this problem, while not wasting bandwidth, is to issue a unilateral FETCH FLAGS response whenever a flag changes. There doesn't seem to be a way to get dovecot (I'm using 0.99.10) to do this. Can the 1.0-test versions do this?
Dovecot has done this always. Most clients are just too stupid to do anything with them.
I don't think this is true. I've included the relevant parts IMAP session below while doing the following:
- Sent an email to myself.
- Waited for client A (evolution) to do a SELECT INBOX.
- Went over to client B (squirrelmail) and read the new message.
- Waited for client A to do another SELECT INBOX or similar.
at this point I expect client A to change the status of the mail in question to SEEN, but it stays as UNSEEN. The session below is from client A. Command A00069 reports [UNSEEN 16265] and command A00077 reports 0 unseen messages, so client B clearly read the message between these two points, and the SEEN flag has changed. Yet no FETCH FLAGS response indicates that change. In addition to the STATUS (UNSEEN 0) response, there should also be a FETCH FLAGS response under command A00077 which lets client A know that message 16265 has been seen.
A00057 SELECT INBOX
- FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
- OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft *)]
Flags permitted.
- 16265 EXISTS
- 1 RECENT
- OK [UNSEEN 16265] First unseen.
- OK [UIDVALIDITY 1095611075] UIDs valid
- OK [UIDNEXT 18918] Predicted next UID
A00057 OK [READ-WRITE] Select completed. A00058 FETCH 16264 UID
- 16264 FETCH (UID 18916)
A00058 OK Fetch completed. A00059 UID FETCH 18917:* (FLAGS RFC822.SIZE INTERNALDATE BODY.PEEK [HEADER.FIELDS.NOT (RECEIVED)])
- 16265 FETCH (UID 18917 FLAGS (\Recent) INTERNALDATE "29-Nov-2004
21:19:54 -0500" RFC822.SIZE 1043 BODY[HEADER.FIELDS.NOT (RECEIVED)] {467} ) A00059 OK Fetch completed. A00069 SELECT INBOX
- FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
- OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft *)]
Flags permitted.
- 16265 EXISTS
- 0 RECENT
- OK [UNSEEN 16265] First unseen.
- OK [UIDVALIDITY 1095611075] UIDs valid
- OK [UIDNEXT 18918] Predicted next UID
A00069 OK [READ-WRITE] Select completed. A00070 FETCH 16265 UID
- 16265 FETCH (UID 18917)
A00070 OK Fetch completed. A00071 UID FETCH 18917 BODY.PEEK[]
- 16265 FETCH (UID 18917 BODY[] {1043}
) A00071 OK Fetch completed. A00077 STATUS INBOX (MESSAGES UNSEEN)
- STATUS "INBOX" (MESSAGES 16265 UNSEEN 0)
A00077 OK Status completed. A00078 SELECT INBOX
- FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
- OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft *)]
Flags permitted.
- 16265 EXISTS
- 0 RECENT
- OK [UIDVALIDITY 1095611075] UIDs valid
- OK [UIDNEXT 18918] Predicted next UID
A00078 OK [READ-WRITE] Select completed. A00079 FETCH 16265 UID
- 16265 FETCH (UID 18917)
A00079 OK Fetch completed.
On 30.11.2004, at 05:07, Aaron VanDevender wrote:
I don't think this is true. I've included the relevant parts IMAP session below while doing the following:
- Sent an email to myself.
- Waited for client A (evolution) to do a SELECT INBOX.
- Went over to client B (squirrelmail) and read the new message.
- Waited for client A to do another SELECT INBOX or similar.
at this point I expect client A to change the status of the mail in question to SEEN, but it stays as UNSEEN. The session below is from client A. Command A00069 reports [UNSEEN 16265]
That does seem to be bug.
and command A00077 reports 0 unseen messages, so client B clearly read the message between these two points, and the SEEN flag has changed. Yet no FETCH FLAGS response indicates that change. In addition to the STATUS (UNSEEN 0) response, there should also be a FETCH FLAGS response under command A00077 which lets client A know that message 16265 has been seen.
If Evolution had done NOOP command instead of SELECT, it would have received the FETCH FLAGS response. I don't think the FETCH FLAGS should be sent when mailbox is being changed (even if it's for the same mailbox).
In any case, last I checked Evolution (v1.4.x) just ignored FETCH FLAGS replies, so even if Dovecot did send them it wouldn't be of any use to you.
On Tue, 2004-11-30 at 09:00 +0200, Timo Sirainen wrote:
at this point I expect client A to change the status of the mail in question to SEEN, but it stays as UNSEEN. The session below is from client A. Command A00069 reports [UNSEEN 16265]
That does seem to be bug.
Do you mean an Evolution bug, or a Dovecot bug?
If Evolution had done NOOP command instead of SELECT, it would have received the FETCH FLAGS response. I don't think the FETCH FLAGS should be sent when mailbox is being changed (even if it's for the same mailbox).
I think it should give FETCH FLAGS responses to a SELECT request, and I think that RFC2060 agrees with me, although it's pretty ambiguous for the non-NOOP case. But my interpretation of section 7 is that unilateral FETCH FLAGS updates should come whenever a flag changes, no matter what the client is doing. The reason is that even if you are giving periodic NOOP requests, from time to time client A will still want to FETCH a different folder or write something to a Sent mail folder or something, and its possible that a flag could get updated on a message in the interval after the last NOOP, but before the client can SELECT back to the INBOX folder, meaning the client can still get out of sync, even if it is sending NOOP requests constantly. The only way to protect against that sort of race condition is to have the SELECT request return FETCH updates.
This race window is even wider for other folders. If the client spends most of its time in INBOX, and only periodically SELECTs over to, say, a Mailing-List folder, for a STATUS or a LIST check, then most likely if a message in the Mailing-List folder is marked SEEN it won't be picked up by the client at all, because it was NOOPing INBOX and SELECT doesn't return FETCH FLAGS. The only way to avoid this is just to have SELECT return FETCH FLAGS updates.
In any case, last I checked Evolution (v1.4.x) just ignored FETCH FLAGS replies, so even if Dovecot did send them it wouldn't be of any use to you.
I'm using Evolution 2.0.2 and the evo IMAP support has been rewritten a couple of times since 1.4.x. (and is destined to be rewritten at least once or twice more in the near future; perhaps IMAP isn't the easiest spec to implement reliably ;-) I think Evo does respond to FETCH FLAGS now, so it might actually be of use. If it doesn't I can start writing patches for Evolution, but there's no point really if the server doesn't give them. This is a little bit of a chicken-and-egg problem, but it seems like starting with the server would be easier, so why not start there, since there's a good chance the client is already done?
On a related note about Evolution's IMAP support, there is a comment in dovecot.conf in reference to mailbox_check_interval that says:
# NOTE: Evolution client breaks with this option when it's trying to APPEND.
This has been fixed for over a year. http://bugzilla.ximian.com/show_bug.cgi?id=42573
cya, .sig
On 1.12.2004, at 00:30, Aaron VanDevender wrote:
On Tue, 2004-11-30 at 09:00 +0200, Timo Sirainen wrote:
at this point I expect client A to change the status of the mail in question to SEEN, but it stays as UNSEEN. The session below is from client A. Command A00069 reports [UNSEEN 16265]
That does seem to be bug.
Do you mean an Evolution bug, or a Dovecot bug?
Dovecot bug.
This race window is even wider for other folders. If the client spends most of its time in INBOX, and only periodically SELECTs over to, say, a Mailing-List folder, for a STATUS or a LIST check, then most likely if a message in the Mailing-List folder is marked SEEN it won't be picked up by the client at all, because it was NOOPing INBOX and SELECT doesn't return FETCH FLAGS. The only way to avoid this is just to have SELECT return FETCH FLAGS updates.
In theory that could work for a single session, but I don't think it's such a good idea. It assumes that all clients want to know about all changes all the time. This isn't true for example for Pine, webmail+caching imap proxy, and other clients that just don't care and fetch only what they want to know.
And most of the clients that do want to know about all changes issue FETCH 1:* FLAGS right after SELECT in any case, making Dovecot only waste bandwidth with them. Evolution is about the only exception which doesn't do this.
Also if the connection gets closed, there is no longer any way to resend the flags because there's no way to identify one client from another.
The real solution for this is CONDSTORE extension which allows fetching only changed flags.
In any case, last I checked Evolution (v1.4.x) just ignored FETCH FLAGS replies, so even if Dovecot did send them it wouldn't be of any use to you.
I'm using Evolution 2.0.2 and the evo IMAP support has been rewritten a couple of times since 1.4.x. (and is destined to be rewritten at least once or twice more in the near future; perhaps IMAP isn't the easiest spec to implement reliably ;-)
Last I heard 2.0 has the same old IMAP code that has been there forever with only some minor fixes, but the rewritten IMAP code is included in the source tarball, just not built by default. Too bad it still doesn't have IDLE support.
I think Evo does respond to FETCH FLAGS now, so it might actually be of use. If it doesn't I can start writing patches for Evolution, but there's no point really if the server doesn't give them. This is a little bit of a chicken-and-egg problem, but it seems like starting with the server would be easier, so why not start there, since there's a good chance the client is already done?
Dovecot behaves the same way as all other servers. It shows the changes for selected mailbox after each command. I think changing both of them to support CONDSTORE would be better idea.
On a related note about Evolution's IMAP support, there is a comment in dovecot.conf in reference to mailbox_check_interval that says:
# NOTE: Evolution client breaks with this option when it's trying to APPEND.
This has been fixed for over a year. http://bugzilla.ximian.com/show_bug.cgi?id=42573
Yes, but Dovecot 0.99.x codebase also is over a year old mostly and I try to avoid changing it much. In any case mailbox_check_interval is quite useless and I've removed it since. The clients that would be able to use it are the ones that support IDLE anyway.
Timo,
(Sorry for the delay in replying. I had a...thing.)
On Wed, 2004-12-01 at 00:54 +0200, Timo Sirainen wrote:
On 1.12.2004, at 00:30, Aaron VanDevender wrote:
On Tue, 2004-11-30 at 09:00 +0200, Timo Sirainen wrote:
at this point I expect client A to change the status of the mail in question to SEEN, but it stays as UNSEEN. The session below is from client A. Command A00069 reports [UNSEEN 16265]
That does seem to be bug.
Do you mean an Evolution bug, or a Dovecot bug?
Dovecot bug.
Is this a dovecot bug that might be getting a fix anytime soon?
This race window is even wider for other folders. If the client spends most of its time in INBOX, and only periodically SELECTs over to, say, a Mailing-List folder, for a STATUS or a LIST check, then most likely if a message in the Mailing-List folder is marked SEEN it won't be picked up by the client at all, because it was NOOPing INBOX and SELECT doesn't return FETCH FLAGS. The only way to avoid this is just to have SELECT return FETCH FLAGS updates.
In theory that could work for a single session, but I don't think it's such a good idea. It assumes that all clients want to know about all changes all the time. This isn't true for example for Pine, webmail+caching imap proxy, and other clients that just don't care and fetch only what they want to know.
But pine doesn't constantly SELECT between folders either, and as long as it dosen't do that, its not going to get flag updates for folders it isn't interested in. I don't think that caching imap proxies are doing a lot of automatic SELECTs either.
And most of the clients that do want to know about all changes issue FETCH 1:* FLAGS right after SELECT in any case, making Dovecot only waste bandwidth with them. Evolution is about the only exception which doesn't do this.
But isn't FETCH 1:* FLAGS the *real* bandwidth culprit? I mean which is going to cost you more bandwidth: 1) requesting the flags of 10k messages every minute continuously, or 2) simply being notified of updated flags, and only updated flags, only when something changes? Clearly 2 is going to use vastly less bandwidth. If we cater to style 2, we might waste a tiny fraction of our bandwidth for some clients, but evolution will save a bundle (and other clients might adopt that strategy), and everyone's clients will work correctly. If we stick with style 1, we save a bit of bandwidth on some people's clients and Evolution's flags gets broken. The choice seems pretty clear to me. (And didn't you admit that this was a bug in dovecot at the top of this mail?)
Also if the connection gets closed, there is no longer any way to resend the flags because there's no way to identify one client from another.
I think that any client which drops the connection and reopens it is going to have to resync. Evolution already does this. But I don't see what that has to do with SELECTing different folders.
The real solution for this is CONDSTORE extension which allows fetching only changed flags.
Yes, well I don't think that solution is going to get us out of any holes anytime soon.
On Thu, 2005-03-24 at 14:57 -0600, Aaron VanDevender wrote:
at this point I expect client A to change the status of the mail in question to SEEN, but it stays as UNSEEN. The session below is from client A. Command A00069 reports [UNSEEN 16265]
That does seem to be bug.
Do you mean an Evolution bug, or a Dovecot bug?
Dovecot bug.
Is this a dovecot bug that might be getting a fix anytime soon?
It should be fixed in 1.0-tests/stable. Most likely won't get fixed in 0.99.x..
This race window is even wider for other folders. If the client spends most of its time in INBOX, and only periodically SELECTs over to, say, a Mailing-List folder, for a STATUS or a LIST check, then most likely if a message in the Mailing-List folder is marked SEEN it won't be picked up by the client at all, because it was NOOPing INBOX and SELECT doesn't return FETCH FLAGS. The only way to avoid this is just to have SELECT return FETCH FLAGS updates.
In theory that could work for a single session, but I don't think it's such a good idea. It assumes that all clients want to know about all changes all the time. This isn't true for example for Pine, webmail+caching imap proxy, and other clients that just don't care and fetch only what they want to know.
But pine doesn't constantly SELECT between folders either, and as long as it dosen't do that, its not going to get flag updates for folders it isn't interested in. I don't think that caching imap proxies are doing a lot of automatic SELECTs either.
But when you actually change the folder in Pine, it starts fetching the flags for the messages that are visible in screen. So depending on the height of your terminal, something like 20-50.
And most of the clients that do want to know about all changes issue FETCH 1:* FLAGS right after SELECT in any case, making Dovecot only waste bandwidth with them. Evolution is about the only exception which doesn't do this.
But isn't FETCH 1:* FLAGS the *real* bandwidth culprit?
Yes. But that's what most of the clients do. I can't change that.
I mean which is going to cost you more bandwidth: 1) requesting the flags of 10k messages every minute continuously, or 2) simply being notified of updated flags, and only updated flags, only when something changes? Clearly 2 is going to use vastly less bandwidth. If we cater to style 2, we might waste a tiny fraction of our bandwidth for some clients, but evolution will save a bundle (and other clients might adopt that strategy), and everyone's clients will work correctly. If we stick with style 1, we save a bit of bandwidth on some people's clients and Evolution's flags gets broken. The choice seems pretty clear to me. (And didn't you admit that this was a bug in dovecot at the top of this mail?)
No, just the UNSEEN sequence number was wrong.
Also if the connection gets closed, there is no longer any way to resend the flags because there's no way to identify one client from another.
I think that any client which drops the connection and reopens it is going to have to resync. Evolution already does this. But I don't see what that has to do with SELECTing different folders.
So, you meant the flags would only be sent between SELECTs within same session. Hmm. That might work. Only problem is that Dovecot would use up more memory since it has to keep all the accessed mailboxes open. Not a really good idea.
Also most of the clients wouldn't want to do that because Dovecot would be the only server where it worked. So all in all, this would just benefit Evolution users.
The real solution for this is CONDSTORE extension which allows fetching only changed flags.
Yes, well I don't think that solution is going to get us out of any holes anytime soon.
It's pretty much the only non-kludgy solution.
participants (2)
-
Aaron VanDevender
-
Timo Sirainen