[Dovecot] Mail.app IDLE support
Hi,
I'm trying to use the new IDLE support in Mail.app on Mac OS 10.5, and
it works sort of, but it seems to miss some notifications.
My guess is the IDLE support in Mail.app is implemented such that it
sets up a connection for IDLE notifications and then checks the mail
via another connection, and because of that the "IDLE-connection" gets
disconnected for inactivity after 30 min. The reason for my guess is
that I get the following in my logs:
dovecot: Nov 14 17:00:29 Info: imap-login: Login: user=<amanda>,
method=PLAIN, rip=::ffff:xxx.xxx.xxx.xxx, lip=::ffff:192.168.1.2, TLS
dovecot: Nov 14 17:00:29 Info: imap-login: Login: user=<ehmsen>,
method=PLAIN, rip=::ffff:xxx.xxx.xxx.xxx, lip=::ffff:192.168.1.2, TLS
dovecot: Nov 14 17:00:30 Info: imap-login: Login: user=<amanda>,
method=PLAIN, rip=::ffff:xxx.xxx.xxx.xxx, lip=::ffff:192.168.1.2, TLS
dovecot: Nov 14 17:00:50 Info: imap-login: Login: user=<ehmsen>,
method=PLAIN, rip=::ffff:xxx.xxx.xxx.xxx, lip=::ffff:192.168.1.2, TLS
dovecot: Nov 14 17:00:51 Info: imap-login: Login: user=<ehmsen>,
method=PLAIN, rip=::ffff:xxx.xxx.xxx.xxx, lip=::ffff:192.168.1.2, TLS
dovecot: Nov 14 17:04:58 Info: imap-login: Login: user=<ehmsen>,
method=PLAIN, rip=::ffff:xxx.xxx.xxx.xxx, lip=::ffff:192.168.1.2, TLS
dovecot: Nov 14 17:30:40 Info: IMAP(amanda): Disconnected for inactivity
dovecot: Nov 14 17:43:29 Info: IMAP(amanda): Disconnected for inactivity
Is there something I can do about this problem? Is the IDLE
implementation in the 1.1 branch different from the implementation in
the 1.0 branch of dovecot?
Or can I debug this issue further?
I guess these things are standardized, but it seems hard for Apple to
follow a standard :-)
Thanks in advance, Martin R. Ehmsen
On Wed, 2007-11-14 at 13:05 -0500, Martin R. Ehmsen wrote:
I'm trying to use the new IDLE support in Mail.app on Mac OS 10.5, and
it works sort of, but it seems to miss some notifications. My guess is the IDLE support in Mail.app is implemented such that it
sets up a connection for IDLE notifications and then checks the mail
via another connection, and because of that the "IDLE-connection" gets
disconnected for inactivity after 30 min. The reason for my guess is
that I get the following in my logs: .. dovecot: Nov 14 17:43:29 Info: IMAP(amanda): Disconnected for inactivityIs there something I can do about this problem? Is the IDLE
implementation in the 1.1 branch different from the implementation in
the 1.0 branch of dovecot?
Yes, it's different there. In v1.1 it never actually disconnects for inactivity. It sends "Still here" notifications every 2 minutes, so if the connection is really dead those sends should cause Dovecot to notice the disconnection.
But is disconnection the real problem? I changed IDLE a bit also in v1.0.6:
* IDLE: Interval between mailbox change notifies is now 1 second,
because some clients keep a long-running IDLE connection and use
other connections to actually read the mails.On 14/11/2007, at 13.13, Timo Sirainen wrote:
On Wed, 2007-11-14 at 13:05 -0500, Martin R. Ehmsen wrote:
I'm trying to use the new IDLE support in Mail.app on Mac OS 10.5,
and it works sort of, but it seems to miss some notifications. My guess is the IDLE support in Mail.app is implemented such that it sets up a connection for IDLE notifications and then checks the mail via another connection, and because of that the "IDLE-connection"
gets disconnected for inactivity after 30 min. The reason for my guess is that I get the following in my logs: .. dovecot: Nov 14 17:43:29 Info: IMAP(amanda): Disconnected for
inactivityIs there something I can do about this problem? Is the IDLE implementation in the 1.1 branch different from the implementation in the 1.0 branch of dovecot?
Yes, it's different there. In v1.1 it never actually disconnects for inactivity. It sends "Still here" notifications every 2 minutes, so if the connection is really dead those sends should cause Dovecot to
notice the disconnection.But is disconnection the real problem? I changed IDLE a bit also in v1.0.6:
- IDLE: Interval between mailbox change notifies is now 1 second, because some clients keep a long-running IDLE connection and use other connections to actually read the mails.
I'm not sure that it is disconnections that is the real problem, that
was just what I figured from the above log.
Is there some way to get more verbose output in the logs that could
help debug my problem? (the only verbose options are for auth and ssl,
and no options seems to apply under the log section of the config file).
I'm using v. 1.0.7, but I guess I could try the 1.1 branch even though
it is in beta :-)
Martin
On 14.11.2007 21:26, Martin R. Ehmsen wrote:
On 14/11/2007, at 13.13, Timo Sirainen wrote:
On Wed, 2007-11-14 at 13:05 -0500, Martin R. Ehmsen wrote:
I'm trying to use the new IDLE support in Mail.app on Mac OS 10.5, and it works sort of, but it seems to miss some notifications. My guess is the IDLE support in Mail.app is implemented such that it sets up a connection for IDLE notifications and then checks the mail via another connection, and because of that the "IDLE-connection" gets disconnected for inactivity after 30 min. The reason for my guess is that I get the following in my logs: .. dovecot: Nov 14 17:43:29 Info: IMAP(amanda): Disconnected for inactivity
Is there something I can do about this problem? Is the IDLE implementation in the 1.1 branch different from the implementation in the 1.0 branch of dovecot?
Yes, it's different there. In v1.1 it never actually disconnects for inactivity. It sends "Still here" notifications every 2 minutes, so if the connection is really dead those sends should cause Dovecot to notice the disconnection.
But is disconnection the real problem? I changed IDLE a bit also in v1.0.6:
* IDLE: Interval between mailbox change notifies is now 1 second, because some clients keep a long-running IDLE connection and use other connections to actually read the mails.I'm not sure that it is disconnections that is the real problem, that was just what I figured from the above log. Is there some way to get more verbose output in the logs that could help debug my problem? (the only verbose options are for auth and ssl, and no options seems to apply under the log section of the config file). I'm using v. 1.0.7, but I guess I could try the 1.1 branch even though it is in beta :-)
Martin I think most easy way is just tcpdump packets from client to see whats going on.
On 14/11/2007, at 13.29, Nikolay Shopik wrote:
I'm not sure that it is disconnections that is the real problem,
that was just what I figured from the above log. Is there some way to get more verbose output in the logs that could
help debug my problem? (the only verbose options are for auth and
ssl, and no options seems to apply under the log section of the
config file). I'm using v. 1.0.7, but I guess I could try the 1.1 branch even
though it is in beta :-)Martin I think most easy way is just tcpdump packets from client to see
whats going on.
Any hint on how to use tcpdump to only grab the interesting parts?
My knowledge of tcpdump is very limited and I only seem to be able to
get a lot of noise out of it.
Martin
Any hint on how to use tcpdump to only grab the interesting parts?
tcpdump port 143
or whatever.
My knowledge of tcpdump is very limited and I only seem to be able to get a lot of noise out of it.
Look at ngrep - it lets you use both tcpdump expressions, *and* further filter by regular expression. And, it shows ascii output instead of all the hex crap - makes it a lot easier to follow ascii based protocols.
Either way, you'll want to disable SSL on the client, so that you can see the traffic properly. :-)
On Wed, 14 Nov 2007, Jason Fesler wrote:
Any hint on how to use tcpdump to only grab the interesting parts?
tcpdump port 143
or whatever.
My knowledge of tcpdump is very limited and I only seem to be able to get a lot of noise out of it.
Look at ngrep - it lets you use both tcpdump expressions, *and* further filter by regular expression. And, it shows ascii output instead of all the hex crap - makes it a lot easier to follow ascii based protocols.
Either way, you'll want to disable SSL on the client, so that you can see the traffic properly. :-)
Though I've not tried ngrep (Thanks!), I've found Wireshark (formerly Ethereal) to be incredibly easy to use for these types of tasks.
In Wireshark, you'd just start a capture and filter it to:
protocol = imap
Best, Ben
On 15.11.2007 0:05, Benjamin R. Haskell wrote:
On Wed, 14 Nov 2007, Jason Fesler wrote:
Any hint on how to use tcpdump to only grab the interesting parts?
tcpdump port 143
or whatever.
My knowledge of tcpdump is very limited and I only seem to be able to get a lot of noise out of it.
Look at ngrep - it lets you use both tcpdump expressions, *and* further filter by regular expression. And, it shows ascii output instead of all the hex crap - makes it a lot easier to follow ascii based protocols.
Either way, you'll want to disable SSL on the client, so that you can see the traffic properly. :-)
Though I've not tried ngrep (Thanks!), I've found Wireshark (formerly Ethereal) to be incredibly easy to use for these types of tasks.
In Wireshark, you'd just start a capture and filter it to:
protocol = imap
Best, Ben
Installing Wireshark on Mac OS X is bit tricky even installation on linux simpler, also its good idea using wireshark to analyze tcpdump dumps from server. Make analyzing work much faster.
On 14/11/2007, at 14.21, Jason Fesler wrote:
Any hint on how to use tcpdump to only grab the interesting parts?
tcpdump port 143
or whatever.
My knowledge of tcpdump is very limited and I only seem to be able
to get a lot of noise out of it.Look at ngrep - it lets you use both tcpdump expressions, *and*
further filter by regular expression. And, it shows ascii output
instead of all the hex crap - makes it a lot easier to follow ascii
based protocols.Either way, you'll want to disable SSL on the client, so that you
can see the traffic properly. :-)
I installed dovecot 1.0.7 on my leopard laptop, disabled SSL and
started it on the loopback device.
Then I connected to it from Mail.app using IMAP and the IDLE option in
Mail.app, while running:
tcpdump -A -s 0 -i lo0 'tcp port imap and (((ip[2:2] -
((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)' > dump.txt
Here is what initially came out of the dump:
<cut a lot of stuff>
14:46:52.589999 IP localhost.50770 > localhost.imap: P 394:404(10) ack
1237 win 65535 <nop,nop,timestamp 497198901 497198901>
E..>..@.@............R...V?2(k.......2.....
...5...57.2 IDLE
14:46:52.590346 IP localhost.imap > localhost.50770: P 1237:1247(10)
ack 404 win 65535 <nop,nop,timestamp 497198901 497198901>
E..> .@.@..............R(k...V?<.....2.....
...5...5+ idling
14:46:57.448879 IP localhost.imap > localhost.50777: P
355780982:355781003(21) ack 2104750877 win 65535 <nop,nop,timestamp
497198950 497198950>
E..I..@.@..............Y.4.v}s.......=.....
...f...f* OK Dovecot ready.
14:46:57.459450 IP localhost.50777 > localhost.imap: P 1:17(16) ack 21
win 65535 <nop,nop,timestamp 497198950 497198950>
E..D..@.@............Y..}s...4.......8.....
...f...f1.6 CAPABILITY
14:46:57.459654 IP localhost.imap > localhost.50777: P 21:187(166) ack
17 win 65535 <nop,nop,timestamp 497198950 497198950>
E.....@.@..............Y.4..}s.-...........
...f...f* CAPABILITY IMAP4rev1 SASL-IR SORT THREAD=REFERENCES
MULTIAPPEND UNSELECT LITERAL+ IDLE CHILDREN NAMESPACE LOGIN-REFERRALS
AUTH=PLAIN
1.6 OK Capability completed.
<and a lot of more stuff and finally it came to a halt>
14:46:57.542907 IP localhost.imap > localhost.50772: P 449:525(76) ack
209 win 65535 <nop,nop,timestamp 497198951 497198951>
E...w.@.@..............T.E....o`.....t.....
...g...g* STATUS "Deleted Messages" (UIDNEXT 3 UNSEEN 0)
7.3 OK Status completed.
14:48:52.590857 IP localhost.imap > localhost.50770: P 1247:1264(17)
ack 404 win 65535 <nop,nop,timestamp 497200094 497198901>
E..EI.@.@..............R(k...V?<.....9.....
.......5* OK Still here
14:50:52.590661 IP localhost.imap > localhost.50770: P 1264:1281(17)
ack 404 win 65535 <nop,nop,timestamp 497201286 497200094>
E..E..@.@..............R(k...V?<.....9.....
........* OK Still here
14:52:52.590964 IP localhost.imap > localhost.50770: P 1281:1298(17)
ack 404 win 65535 <nop,nop,timestamp 497202478 497201286>
E..E..@.@..............R(k...V?<.....9.....
........* OK Still here
The above shows the only time IDLE is every used (except when Mail.app
asks for the capabilities of the server).
Somewhere after 14:46:57, but before 14:48:52, I copied (since I have
no SMTP server running on my laptop) a new mail into the new directory
of the maildir I subscribed to. But as you can see from the above,
somehow dovecot failed to notice the new mail.
I don't use (d/i)notify or kqueue, but I did set dovecot up to check
every 30 sec, which it obviously didn't.
Btw. the mail arrives fine if I manually check for new mail in Mail.app.
I can provide the full dump above if anybody thinks it is going to be
useful?
Martin
On 15.11.2007, at 22.08, Martin R. Ehmsen wrote:
The above shows the only time IDLE is every used (except when
Mail.app asks for the capabilities of the server). Somewhere after 14:46:57, but before 14:48:52, I copied (since I
have no SMTP server running on my laptop) a new mail into the new
directory of the maildir I subscribed to. But as you can see from
the above, somehow dovecot failed to notice the new mail. I don't use (d/i)notify or kqueue, but I did set dovecot up to
check every 30 sec, which it obviously didn't. Btw. the mail arrives fine if I manually check for new mail in
Mail.app.
IDLE checks for new mails only in the selected mailbox, not all
mailboxes. IMAP doesn't really support that.
On 15.11.2007 23:08, Martin R. Ehmsen wrote:
On 14/11/2007, at 14.21, Jason Fesler wrote:
Any hint on how to use tcpdump to only grab the interesting parts?
tcpdump port 143
or whatever.
My knowledge of tcpdump is very limited and I only seem to be able to get a lot of noise out of it.
Look at ngrep - it lets you use both tcpdump expressions, *and* further filter by regular expression. And, it shows ascii output instead of all the hex crap - makes it a lot easier to follow ascii based protocols.
Either way, you'll want to disable SSL on the client, so that you can see the traffic properly. :-)
I installed dovecot 1.0.7 on my leopard laptop, disabled SSL and started it on the loopback device. Then I connected to it from Mail.app using IMAP and the IDLE option in Mail.app, while running:
tcpdump -A -s 0 -i lo0 'tcp port imap and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)' > dump.txt
Here is what initially came out of the dump:
<cut a lot of stuff>
14:46:52.589999 IP localhost.50770 > localhost.imap: P 394:404(10) ack 1237 win 65535 <nop,nop,timestamp 497198901 497198901> E..>..@.@............R...V?2(k.......2..... ...5...57.2 IDLE
14:46:52.590346 IP localhost.imap > localhost.50770: P 1237:1247(10) ack 404 win 65535 <nop,nop,timestamp 497198901 497198901> E..> .@.@..............R(k...V?<.....2..... ...5...5+ idling
14:46:57.448879 IP localhost.imap > localhost.50777: P 355780982:355781003(21) ack 2104750877 win 65535 <nop,nop,timestamp 497198950 497198950> E..I..@.@..............Y.4.v}s.......=..... ...f...f* OK Dovecot ready.
14:46:57.459450 IP localhost.50777 > localhost.imap: P 1:17(16) ack 21 win 65535 <nop,nop,timestamp 497198950 497198950> E..D..@.@............Y..}s...4.......8..... ...f...f1.6 CAPABILITY
14:46:57.459654 IP localhost.imap > localhost.50777: P 21:187(166) ack 17 win 65535 <nop,nop,timestamp 497198950 497198950> E.....@.@..............Y.4..}s.-........... ...f...f* CAPABILITY IMAP4rev1 SASL-IR SORT THREAD=REFERENCES MULTIAPPEND UNSELECT LITERAL+ IDLE CHILDREN NAMESPACE LOGIN-REFERRALS AUTH=PLAIN 1.6 OK Capability completed.
<and a lot of more stuff and finally it came to a halt>
14:46:57.542907 IP localhost.imap > localhost.50772: P 449:525(76) ack 209 win 65535 <nop,nop,timestamp 497198951 497198951> E...w.@.@..............T.E....o`.....t..... ...g...g* STATUS "Deleted Messages" (UIDNEXT 3 UNSEEN 0) 7.3 OK Status completed.
14:48:52.590857 IP localhost.imap > localhost.50770: P 1247:1264(17) ack 404 win 65535 <nop,nop,timestamp 497200094 497198901> E..EI.@.@..............R(k...V?<.....9..... .......5* OK Still here
14:50:52.590661 IP localhost.imap > localhost.50770: P 1264:1281(17) ack 404 win 65535 <nop,nop,timestamp 497201286 497200094> E..E..@.@..............R(k...V?<.....9..... ........* OK Still here
14:52:52.590964 IP localhost.imap > localhost.50770: P 1281:1298(17) ack 404 win 65535 <nop,nop,timestamp 497202478 497201286> E..E..@.@..............R(k...V?<.....9..... ........* OK Still here
The above shows the only time IDLE is every used (except when Mail.app asks for the capabilities of the server). Somewhere after 14:46:57, but before 14:48:52, I copied (since I have no SMTP server running on my laptop) a new mail into the new directory of the maildir I subscribed to. But as you can see from the above, somehow dovecot failed to notice the new mail. I don't use (d/i)notify or kqueue, but I did set dovecot up to check every 30 sec, which it obviously didn't. Btw. the mail arrives fine if I manually check for new mail in Mail.app.
I can provide the full dump above if anybody thinks it is going to be useful? Martin
Mail client usually should open few connection to mail server if you have many dirs.
participants (5)
-
Benjamin R. Haskell
-
Jason Fesler
-
Martin R. Ehmsen
-
Nikolay Shopik
-
Timo Sirainen