[Dovecot] IMAP IDLE, Virtual mailboxes
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Hi,
K-9 Mail (on Android) has been mentioned on this list multiple times recently, I think it's definitely an IMAP client "on the rise". :) It supports IMAP IDLE, and so brings "push mail" to Android mobile phone users. To bring this into the context of this mailing list: I guess a lot of people on this list are making use of server-side filtering (using sieve), so that mails from certain persons or mailing lists end up in the right mailboxes, right on delivery. Now, this is all great, but there are issues when server-side filtering and IMAP IDLE come together: Only one mailbox can be "watched" with one IMAP IDLE connection. So, people who have set up elaborate server-side filtering with say, 50 different sieve "fileinto" calls, will constantly have a *lot* of IMAP IDLE connections open, if they really want to get informed of any mail arriving in any of their mailboxes.
Keeping those 50 IMAP connections open wouldn't cost much bandwidth, and who cares about the Internet router's connection tables getting unnecessarily big, anyway? But then there is the "problem" of a mobile phone being carried around constantly, switching physical connections (from WLAN access point 1 to WLAN access point 2 to 3G to 2G to no connection at all and back), every time needing to terminate* those 50 IMAP connections, and then establish 50 new ones (with SSL handshake overhead and all). So, all in all, not an optimal solution.
*) (which does not always happen the way it should, see http://code.google.com/p/k9mail/issues/detail?id=1255)
I see two ways to improve the situation (one of them quite dovecot-centric):
- A new extension of the IMAP protocol: IMAP MULTI-IDLE, which somehow
allows a client to get notified of changes to more than one mailbox within one IMAP connection. This conflicts with the basic IMAP design though, AFAIK, and is thus not likely to be feasible. 2) Enhancing dovecot's Virtual plugin, so virtual mailboxes do not only get updated on select and expunge, but also when anything changes that affects the set of messages shown in the virtual mailbox. I guess that would have an impact on performance, and thus should be optional.
Let me explain the second one: With the help of the Virtual plugin, it is possible to pull all those 50 mailboxes that receive incoming mail together again, just like the INBOX of the pre-server-side-filtering times - the user could configure a virtual mailbox to show all unread mails from those 50 mailboxes. Now the mobile IMAP IDLE users could just watch that one virtual mailbox, needing only one IMAP IDLE connection. After having read the mail (directly in the virtual mailbox), it would disappear, but of course still be in the "real" mailbox, determined by the sieve filter on delivery.
Patrick.
STAR Software (Shanghai) Co., Ltd. http://www.star-group.net/ Phone: +86 (21) 3462 7688 x 826 Fax: +86 (21) 3462 7779
PGP key E883A005 https://stshacom1.star-china.net/keys/patrick_nagel.asc Fingerprint: E09A D65E 855F B334 E5C3 5386 EF23 20FC E883 A005
On 09.04.2010 10:31, Patrick Nagel wrote:
- A new extension of the IMAP protocol: IMAP MULTI-IDLE, which somehow
allows a client to get notified of changes to more than one mailbox within one IMAP connection. This conflicts with the basic IMAP design though, AFAIK, and is thus not likely to be feasible.
That's RFC5465 IMAP NOTIFY Extension you are talking about and already standards track.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Hi Nikolay,
On 2010-04-09 14:51, Nikolay Shopik wrote:
On 09.04.2010 10:31, Patrick Nagel wrote:
- A new extension of the IMAP protocol: IMAP MULTI-IDLE, which
somehow allows a client to get notified of changes to more than one mailbox within one IMAP connection. This conflicts with the basic IMAP design though, AFAIK, and is thus not likely to be feasible.
That's RFC5465 IMAP NOTIFY Extension you are talking about and already standards track.
That's great news, thanks for the pointer! :)
In http://dovecot.org/list/dovecot/2009-August/041950.html Timo says that IMAP NOTIFY is not yet supported by Dovecot - has this changed? If not, is there a plan to implement IMAP NOTIFY?
Patrick.
STAR Software (Shanghai) Co., Ltd. http://www.star-group.net/ Phone: +86 (21) 3462 7688 x 826 Fax: +86 (21) 3462 7779
PGP key E883A005 https://stshacom1.star-china.net/keys/patrick_nagel.asc Fingerprint: E09A D65E 855F B334 E5C3 5386 EF23 20FC E883 A005
On 09.04.2010 11:19, Patrick Nagel wrote:
That's great news, thanks for the pointer!:)
Inhttp://dovecot.org/list/dovecot/2009-August/041950.html Timo says that IMAP NOTIFY is not yet supported by Dovecot - has this changed? If not, is there a plan to implement IMAP NOTIFY?
Patrick.
Keep in mind it still need at least one connection per mailbox if I correctly understood RFC. Also I don't know any client supporting IMAP NOTIFY yet.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Hi Nikolay,
On Fri, 09 Apr 2010 21:12:54 +0400, Nikolay Shopik wrote:
On 09.04.2010 11:19, Patrick Nagel wrote:
That's great news, thanks for the pointer!:)
Inhttp://dovecot.org/list/dovecot/2009-August/041950.html Timo says that IMAP NOTIFY is not yet supported by Dovecot - has this changed? If not, is there a plan to implement IMAP NOTIFY?
Keep in mind it still need at least one connection per mailbox if I correctly understood RFC.
I don't think so - in chapter 1 (Overview and Rationale) it states "Also, IDLE only applies to the selected mailbox, thus requiring an additional TCP connection per mailbox"
... and later, in chapter 3.1:
"The client can also request notifications on other mailboxes by name or by a limited mailbox pattern match. Message-related notifications returned for the currently selected mailbox will be those specified by the SELECTED/SELECTED-DELAYED mailbox specifier, even if the selected mailbox also appears by name (or matches a pattern) in the command."
I cannot find any indication that multiple IMAP connections are required.
Also I don't know any client supporting IMAP NOTIFY yet.
Well, this is probably a chicken/egg problem ;) One of the core developers of K-9 Mail says: "I've been interested in implementing IMAP NOTIFY in K-9 Mail, but will only do it once dovecot supports it." (http://code.google.com/p/k9mail/issues/detail?id=1255#c8)
Patrick.
STAR Software (Shanghai) Co., Ltd. http://www.star-group.net/ Phone: +86 (21) 3462 7688 x 826 Fax: +86 (21) 3462 7779
PGP key E883A005 https://stshacom1.star-china.net/keys/patrick_nagel.asc Fingerprint: E09A D65E 855F B334 E5C3 5386 EF23 20FC E883 A005
On 10.04.2010 11:13, Patrick Nagel wrote:
"The client can also request notifications on other mailboxes by name or by a limited mailbox pattern match. Message-related notifications returned for the currently selected mailbox will be those specified by the SELECTED/SELECTED-DELAYED mailbox specifier, even if the selected mailbox also appears by name (or matches a pattern) in the command."
I assume this only apply if all these mailboxes are on same server.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Hi Nikolay,
(I changed my dovecot mailing list subscription from my work mail account to my private mail account - I'm the guy who started this thread)
On 2010-04-10 15:22, Nikolay Shopik wrote:
On 10.04.2010 11:13, Patrick Nagel wrote:
"The client can also request notifications on other mailboxes by name or by a limited mailbox pattern match. Message-related notifications returned for the currently selected mailbox will be those specified by the SELECTED/SELECTED-DELAYED mailbox specifier, even if the selected mailbox also appears by name (or matches a pattern) in the command."
I assume this only apply if all these mailboxes are on same server.
That's for sure - "mailbox" in IMAP always refers to the thing that most people call "folder".
But I think it's unlikely that you need push mail for more than a couple of IMAP accounts on different servers. And in any case, there would be no way to accomplish this, with any protocol, without having a connection to all the IMAP servers.
Patrick.
Key ID: 0x86E346D4 http://patrick-nagel.net/key.asc Fingerprint: 7745 E1BE FA8B FBAD 76AB 2BFC C981 E686 86E3 46D4
Patrick Nagel wrote:
One of the core developers of K-9 Mail says: "I've been interested in implementing IMAP NOTIFY in K-9 Mail, but will only do it once dovecot supports it."
Yeah, that was me. :-) I use dovecot as the reference server for all my work on K-9 Mail, and can't think of how to develop the client side support for IMAP NOTIFY until I have a server to support it.
However, I'd consider starting on it if I knew that dovecot would soon have the support, as well.
Dan.
On 15.04.2010 7:03, Daniel I. Applebaum wrote:
Patrick Nagel wrote:
One of the core developers of K-9 Mail says: "I've been interested in implementing IMAP NOTIFY in K-9 Mail, but will only do it once dovecot supports it."
Yeah, that was me. :-) I use dovecot as the reference server for all my work on K-9 Mail, and can't think of how to develop the client side support for IMAP NOTIFY until I have a server to support it.
However, I'd consider starting on it if I knew that dovecot would soon have the support, as well.
Dan.
Hi Dan,
Thanks for K-9! If you don't mind can I ask you little off topic question feel free answer privatly. Does android advertise to app when network goes down(GSM our of range). If it so, when network goes up(like interface goes up?), do you establish new TCP connection to maintain IDLE? Or just purely rely on timeouts in connection?
Thanks
On 9.4.2010, at 9.31, Patrick Nagel wrote:
- Enhancing dovecot's Virtual plugin, so virtual mailboxes do not only
get updated on select and expunge, but also when anything changes that affects the set of messages shown in the virtual mailbox. I guess that would have an impact on performance, and thus should be optional.
Virtual plugin does already work like that. Although there's a bug where messages don't get removed from mailbox always:
- virtual: removed messages don't get expunged unless EXPUNGE is issued in same session. otherwise they get forgotten and never removed.
And NOTIFY isn't implemented yet mainly because it would be annoyingly expensive (as is virtual mailbox when it's created from many real mailboxes). I've been planning on implementing mailbox list indexes (or they already are implemented, but they're buggy) that would make this much cheaper. Basically NOTIFY could just keep watching for changes to dovecot.list.index.log file, and then read what mailbox had changed and how and notify client about it, possibly without even opening the mailbox itself.
Hi Timo,
great hearing from you again, hope the exams went well :)
On 2010-04-13 19:01 UTC Timo Sirainen wrote:
On 9.4.2010, at 9.31, Patrick Nagel wrote:
- Enhancing dovecot's Virtual plugin, so virtual mailboxes do not
only get updated on select and expunge, but also when anything changes that affects the set of messages shown in the virtual mailbox. I guess that would have an impact on performance, and thus should be optional.
Virtual plugin does already work like that. Although there's a bug where messages don't get removed from mailbox always:
- virtual: removed messages don't get expunged unless EXPUNGE is
issued in same session. otherwise they get forgotten and never removed.
Oh, I see. I'll give it another try then, I believe I tried this, but never got notified of new mails in the virtual mailbox through the IMAP IDLE connection.
And NOTIFY isn't implemented yet mainly because it would be annoyingly expensive (as is virtual mailbox when it's created from many real mailboxes). I've been planning on implementing mailbox list indexes (or they already are implemented, but they're buggy) that would make this much cheaper. Basically NOTIFY could just keep watching for changes to dovecot.list.index.log file, and then read what mailbox had changed and how and notify client about it, possibly without even opening the mailbox itself.
That sounds like a good plan - basically having "logwatch" on all IMAP operations :) I'd be happy to test once you have anything to be tested (just need to find a MUA other than telnet+brain that has IMAP NOTIFY implemented first ;) ).
Patrick.
Hi Timo, Patrick,
Virtual plugin does already work like that. Although there's a bug where messages don't get removed from mailbox always:
Oh, I see. I'll give it another try then, I believe I tried this, but never got notified of new mails in the virtual mailbox through the IMAP IDLE connection.
I'm seeing the same thing: IDLE works fine for normal mailboxes, but fails on a virtual mailbox. Using rawlog I can confirm that the IDLE command is received properly, but there is just no notification of the new mail at all:
a0026 IDLE
< + idling < * OK Still here < * OK Still here ... around a hundred more of these ... < * OK Still here < * OK Still here
DONE
< a0026 OK Idle completed.
(The interleaving of the in and out messages is done by me, but I think this should be how it went)
During this idle period, tens of emails were delivered, but nothing shows up.
Any suggestions on debugging this further? I'm running dovecot 1.2.9 from backports.org (but I could compile my own and add some debbuging statements, if that would help and you have suggestions on where to put them).
Gr.
Matthijs
Hi all,
it's been a while since I wrote below email. Any chance someone has some extra debugging suggestions for me?
Gr.
Matthijs
On Thu, Apr 29, 2010 at 10:00:18AM +0200, Matthijs Kooijman wrote:
Hi Timo, Patrick,
Virtual plugin does already work like that. Although there's a bug where messages don't get removed from mailbox always:
Oh, I see. I'll give it another try then, I believe I tried this, but never got notified of new mails in the virtual mailbox through the IMAP IDLE connection.
I'm seeing the same thing: IDLE works fine for normal mailboxes, but fails on a virtual mailbox. Using rawlog I can confirm that the IDLE command is received properly, but there is just no notification of the new mail at all:
a0026 IDLE
< + idling < * OK Still here < * OK Still here ... around a hundred more of these ... < * OK Still here < * OK Still here
DONE
< a0026 OK Idle completed.
(The interleaving of the in and out messages is done by me, but I think this should be how it went)
During this idle period, tens of emails were delivered, but nothing shows up.
Any suggestions on debugging this further? I'm running dovecot 1.2.9 from backports.org (but I could compile my own and add some debbuging statements, if that would help and you have suggestions on where to put them).
Gr.
Matthijs
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Hi Timo,
On 2010-08-04 23:54, Timo Sirainen wrote:
Thanks, that's great! Now dovecot enables people who sort server-side and want IDLE notifications for all those mailboxes, but do not want to keep "many" IDLE connections open. They can now create a virtual mailbox that contains all mail-receiving mailboxes' mail (lets call it vinbox), and put their IDLE connection on that vinbox. Neat.
I'll update to 2.0 ASAP :)
Patrick.
Key ID: 0x86E346D4 http://patrick-nagel.net/key.asc Fingerprint: 7745 E1BE FA8B FBAD 76AB 2BFC C981 E686 86E3 46D4
Hi Timo,
Cool! I've just tested this patch on 1.2 (current hg version), and the patch applies cleanly and seems to work as well. I'm running it right now.
Are there any plans to include this in 1.2 still?
Gr.
Matthijs
On Thu, 2010-08-05 at 12:22 +0200, Matthijs Kooijman wrote:
Hi Timo,
Cool! I've just tested this patch on 1.2 (current hg version), and the patch applies cleanly and seems to work as well. I'm running it right now.
Are there any plans to include this in 1.2 still?
I was planning to avoid adding any changes to v1.2 that aren't bugfixes. (This more like a feature. :)
- Timo Sirainen tss@iki.fi:
On Thu, 2010-08-05 at 12:22 +0200, Matthijs Kooijman wrote:
Hi Timo,
Cool! I've just tested this patch on 1.2 (current hg version), and the patch applies cleanly and seems to work as well. I'm running it right now.
Are there any plans to include this in 1.2 still?
I was planning to avoid adding any changes to v1.2 that aren't bugfixes. (This more like a feature. :)
Oh, come on. You even wrote "/* FIXME: maybe some day */" :)
On Thu, 2010-08-05 at 16:21 +0200, Ralf Hildebrandt wrote:
I was planning to avoid adding any changes to v1.2 that aren't bugfixes. (This more like a feature. :)
Oh, come on. You even wrote "/* FIXME: maybe some day */" :)
Well, fine.. http://hg.dovecot.org/dovecot-1.2/rev/367ce71922bf
But I don't know when 1.2.14 release will come, if ever.
Well, fine.. http://hg.dovecot.org/dovecot-1.2/rev/367ce71922bf
Woohoo, thanks :-)
But I don't know when 1.2.14 release will come, if ever.
We'll see. For now I'm just running a patched version, but this saves me the trouble of recompiling if 1.2.14 every gets released :-)
Gr.
Matthijs
participants (7)
-
Daniel I. Applebaum
-
Matthijs Kooijman
-
Nikolay Shopik
-
Patrick Nagel
-
Patrick Nagel
-
Ralf Hildebrandt
-
Timo Sirainen