[Dovecot] Mail.app IDLE support

newer
[Dovecot] RHEL 5.1 Can't compile...

Martin R. Ehmsen

14 Nov 2007 14 Nov '07

6:05 p.m.

Hi,

I'm trying to use the new IDLE support in Mail.app on Mac OS 10.5, and
it works sort of, but it seems to miss some notifications. My guess is the IDLE support in Mail.app is implemented such that it
sets up a connection for IDLE notifications and then checks the mail
via another connection, and because of that the "IDLE-connection" gets
disconnected for inactivity after 30 min. The reason for my guess is
that I get the following in my logs:

dovecot: Nov 14 17:00:29 Info: imap-login: Login: user=<amanda>,
method=PLAIN, rip=::ffff:xxx.xxx.xxx.xxx, lip=::ffff:192.168.1.2, TLS dovecot: Nov 14 17:00:29 Info: imap-login: Login: user=<ehmsen>,
method=PLAIN, rip=::ffff:xxx.xxx.xxx.xxx, lip=::ffff:192.168.1.2, TLS dovecot: Nov 14 17:00:30 Info: imap-login: Login: user=<amanda>,
method=PLAIN, rip=::ffff:xxx.xxx.xxx.xxx, lip=::ffff:192.168.1.2, TLS dovecot: Nov 14 17:00:50 Info: imap-login: Login: user=<ehmsen>,
method=PLAIN, rip=::ffff:xxx.xxx.xxx.xxx, lip=::ffff:192.168.1.2, TLS dovecot: Nov 14 17:00:51 Info: imap-login: Login: user=<ehmsen>,
method=PLAIN, rip=::ffff:xxx.xxx.xxx.xxx, lip=::ffff:192.168.1.2, TLS dovecot: Nov 14 17:04:58 Info: imap-login: Login: user=<ehmsen>,
method=PLAIN, rip=::ffff:xxx.xxx.xxx.xxx, lip=::ffff:192.168.1.2, TLS dovecot: Nov 14 17:30:40 Info: IMAP(amanda): Disconnected for inactivity dovecot: Nov 14 17:43:29 Info: IMAP(amanda): Disconnected for inactivity

Is there something I can do about this problem? Is the IDLE
implementation in the 1.1 branch different from the implementation in
the 1.0 branch of dovecot? Or can I debug this issue further?

I guess these things are standardized, but it seems hard for Apple to
follow a standard :-)

Thanks in advance, Martin R. Ehmsen

Attachments:

smime.p7s (application/pkcs7-signature — 1.9 KB)

Show replies by date

Timo Sirainen

14 Nov 14 Nov

6:13 p.m.

On Wed, 2007-11-14 at 13:05 -0500, Martin R. Ehmsen wrote:

...

I'm trying to use the new IDLE support in Mail.app on Mac OS 10.5, and
it works sort of, but it seems to miss some notifications. My guess is the IDLE support in Mail.app is implemented such that it
sets up a connection for IDLE notifications and then checks the mail
via another connection, and because of that the "IDLE-connection" gets
disconnected for inactivity after 30 min. The reason for my guess is
that I get the following in my logs: .. dovecot: Nov 14 17:43:29 Info: IMAP(amanda): Disconnected for inactivity

Is there something I can do about this problem? Is the IDLE
implementation in the 1.1 branch different from the implementation in
the 1.0 branch of dovecot?

Yes, it's different there. In v1.1 it never actually disconnects for inactivity. It sends "Still here" notifications every 2 minutes, so if the connection is really dead those sends should cause Dovecot to notice the disconnection.

But is disconnection the real problem? I changed IDLE a bit also in v1.0.6:

* IDLE: Interval between mailbox change notifies is now 1 second,
  because some clients keep a long-running IDLE connection and use
  other connections to actually read the mails.

Martin R. Ehmsen

6:26 p.m.

On 14/11/2007, at 13.13, Timo Sirainen wrote:

...

On Wed, 2007-11-14 at 13:05 -0500, Martin R. Ehmsen wrote:

...
I'm trying to use the new IDLE support in Mail.app on Mac OS 10.5,
and it works sort of, but it seems to miss some notifications. My guess is the IDLE support in Mail.app is implemented such that it sets up a connection for IDLE notifications and then checks the mail via another connection, and because of that the "IDLE-connection"
gets disconnected for inactivity after 30 min. The reason for my guess is that I get the following in my logs: .. dovecot: Nov 14 17:43:29 Info: IMAP(amanda): Disconnected for
inactivity

Is there something I can do about this problem? Is the IDLE implementation in the 1.1 branch different from the implementation in the 1.0 branch of dovecot?

Yes, it's different there. In v1.1 it never actually disconnects for inactivity. It sends "Still here" notifications every 2 minutes, so if the connection is really dead those sends should cause Dovecot to
notice the disconnection.

But is disconnection the real problem? I changed IDLE a bit also in v1.0.6:

IDLE: Interval between mailbox change notifies is now 1 second, because some clients keep a long-running IDLE connection and use other connections to actually read the mails.

I'm not sure that it is disconnections that is the real problem, that
was just what I figured from the above log. Is there some way to get more verbose output in the logs that could
help debug my problem? (the only verbose options are for auth and ssl,
and no options seems to apply under the log section of the config file). I'm using v. 1.0.7, but I guess I could try the 1.1 branch even though
it is in beta :-)

Martin

Nikolay Shopik

6:29 p.m.

On 14.11.2007 21:26, Martin R. Ehmsen wrote:

...

On 14/11/2007, at 13.13, Timo Sirainen wrote:

...
On Wed, 2007-11-14 at 13:05 -0500, Martin R. Ehmsen wrote:

...
I'm trying to use the new IDLE support in Mail.app on Mac OS 10.5, and it works sort of, but it seems to miss some notifications. My guess is the IDLE support in Mail.app is implemented such that it sets up a connection for IDLE notifications and then checks the mail via another connection, and because of that the "IDLE-connection" gets disconnected for inactivity after 30 min. The reason for my guess is that I get the following in my logs: .. dovecot: Nov 14 17:43:29 Info: IMAP(amanda): Disconnected for inactivity

Is there something I can do about this problem? Is the IDLE implementation in the 1.1 branch different from the implementation in the 1.0 branch of dovecot?

Yes, it's different there. In v1.1 it never actually disconnects for inactivity. It sends "Still here" notifications every 2 minutes, so if the connection is really dead those sends should cause Dovecot to notice the disconnection.

But is disconnection the real problem? I changed IDLE a bit also in v1.0.6:
* IDLE: Interval between mailbox change notifies is now 1 second,
  because some clients keep a long-running IDLE connection and use
  other connections to actually read the mails.
I'm not sure that it is disconnections that is the real problem, that was just what I figured from the above log. Is there some way to get more verbose output in the logs that could help debug my problem? (the only verbose options are for auth and ssl, and no options seems to apply under the log section of the config file). I'm using v. 1.0.7, but I guess I could try the 1.1 branch even though it is in beta :-)

Martin I think most easy way is just tcpdump packets from client to see whats going on.

Martin R. Ehmsen

7:05 p.m.

On 14/11/2007, at 13.29, Nikolay Shopik wrote:

...

...
I'm not sure that it is disconnections that is the real problem,
that was just what I figured from the above log. Is there some way to get more verbose output in the logs that could
help debug my problem? (the only verbose options are for auth and
ssl, and no options seems to apply under the log section of the
config file). I'm using v. 1.0.7, but I guess I could try the 1.1 branch even
though it is in beta :-)

Martin I think most easy way is just tcpdump packets from client to see
whats going on.

Any hint on how to use tcpdump to only grab the interesting parts? My knowledge of tcpdump is very limited and I only seem to be able to
get a lot of noise out of it.

Martin

Jason Fesler

7:21 p.m.

...

Any hint on how to use tcpdump to only grab the interesting parts?

tcpdump port 143

or whatever.

...

My knowledge of tcpdump is very limited and I only seem to be able to get a lot of noise out of it.

Look at ngrep - it lets you use both tcpdump expressions, *and* further filter by regular expression. And, it shows ascii output instead of all the hex crap - makes it a lot easier to follow ascii based protocols.

Either way, you'll want to disable SSL on the client, so that you can see the traffic properly. :-)

Benjamin R. Haskell

9:05 p.m.

On Wed, 14 Nov 2007, Jason Fesler wrote:

...

...
Any hint on how to use tcpdump to only grab the interesting parts?

tcpdump port 143

or whatever.

...
My knowledge of tcpdump is very limited and I only seem to be able to get a lot of noise out of it.

Look at ngrep - it lets you use both tcpdump expressions, *and* further filter by regular expression. And, it shows ascii output instead of all the hex crap - makes it a lot easier to follow ascii based protocols.

Either way, you'll want to disable SSL on the client, so that you can see the traffic properly. :-)

Though I've not tried ngrep (Thanks!), I've found Wireshark (formerly Ethereal) to be incredibly easy to use for these types of tasks.

In Wireshark, you'd just start a capture and filter it to:

protocol = imap

Best, Ben

Nikolay Shopik

9:14 p.m.

On 15.11.2007 0:05, Benjamin R. Haskell wrote:

...

On Wed, 14 Nov 2007, Jason Fesler wrote:

...
...
Any hint on how to use tcpdump to only grab the interesting parts?

tcpdump port 143

or whatever.

...
My knowledge of tcpdump is very limited and I only seem to be able to get a lot of noise out of it.

Look at ngrep - it lets you use both tcpdump expressions, *and* further filter by regular expression. And, it shows ascii output instead of all the hex crap - makes it a lot easier to follow ascii based protocols.

Either way, you'll want to disable SSL on the client, so that you can see the traffic properly. :-)

Though I've not tried ngrep (Thanks!), I've found Wireshark (formerly Ethereal) to be incredibly easy to use for these types of tasks.

In Wireshark, you'd just start a capture and filter it to:

protocol = imap

Best, Ben

Installing Wireshark on Mac OS X is bit tricky even installation on linux simpler, also its good idea using wireshark to analyze tcpdump dumps from server. Make analyzing work much faster.

Martin R. Ehmsen

15 Nov 15 Nov

8:08 p.m.

On 14/11/2007, at 14.21, Jason Fesler wrote:

...

...
Any hint on how to use tcpdump to only grab the interesting parts?

tcpdump port 143

or whatever.

...
My knowledge of tcpdump is very limited and I only seem to be able
to get a lot of noise out of it.

Look at ngrep - it lets you use both tcpdump expressions, *and*
further filter by regular expression. And, it shows ascii output
instead of all the hex crap - makes it a lot easier to follow ascii
based protocols.

Either way, you'll want to disable SSL on the client, so that you
can see the traffic properly. :-)

I installed dovecot 1.0.7 on my leopard laptop, disabled SSL and
started it on the loopback device. Then I connected to it from Mail.app using IMAP and the IDLE option in
Mail.app, while running:

tcpdump -A -s 0 -i lo0 'tcp port imap and (((ip[2:2] -
((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)' > dump.txt

Here is what initially came out of the dump:

14:46:52.589999 IP localhost.50770 > localhost.imap: P 394:404(10) ack
1237 win 65535 E..>..@.@............R...V?2(k.......2..... ...5...57.2 IDLE

14:46:52.590346 IP localhost.imap > localhost.50770: P 1237:1247(10)
ack 404 win 65535 E..> .@.@..............R(k...V?<.....2..... ...5...5+ idling

14:46:57.448879 IP localhost.imap > localhost.50777: P
355780982:355781003(21) ack 2104750877 win 65535 497198950 497198950> E..I..@.@..............Y.4.v}s.......=..... ...f...f* OK Dovecot ready.

14:46:57.459450 IP localhost.50777 > localhost.imap: P 1:17(16) ack 21
win 65535 E..D..@.@............Y..}s...4.......8..... ...f...f1.6 CAPABILITY

14:46:57.459654 IP localhost.imap > localhost.50777: P 21:187(166) ack
17 win 65535 E.....@.@..............Y.4..}s.-........... ...f...f* CAPABILITY IMAP4rev1 SASL-IR SORT THREAD=REFERENCES
MULTIAPPEND UNSELECT LITERAL+ IDLE CHILDREN NAMESPACE LOGIN-REFERRALS
AUTH=PLAIN 1.6 OK Capability completed.

14:46:57.542907 IP localhost.imap > localhost.50772: P 449:525(76) ack
209 win 65535 E...w.@.@..............T.E....o`.....t..... ...g...g* STATUS "Deleted Messages" (UIDNEXT 3 UNSEEN 0) 7.3 OK Status completed.

14:48:52.590857 IP localhost.imap > localhost.50770: P 1247:1264(17)
ack 404 win 65535 E..EI.@.@..............R(k...V?<.....9..... .......5* OK Still here

14:50:52.590661 IP localhost.imap > localhost.50770: P 1264:1281(17)
ack 404 win 65535 E..E..@.@..............R(k...V?<.....9..... ........* OK Still here

14:52:52.590964 IP localhost.imap > localhost.50770: P 1281:1298(17)
ack 404 win 65535 E..E..@.@..............R(k...V?<.....9..... ........* OK Still here

The above shows the only time IDLE is every used (except when Mail.app
asks for the capabilities of the server). Somewhere after 14:46:57, but before 14:48:52, I copied (since I have
no SMTP server running on my laptop) a new mail into the new directory
of the maildir I subscribed to. But as you can see from the above,
somehow dovecot failed to notice the new mail. I don't use (d/i)notify or kqueue, but I did set dovecot up to check
every 30 sec, which it obviously didn't. Btw. the mail arrives fine if I manually check for new mail in Mail.app.

I can provide the full dump above if anybody thinks it is going to be
useful?

Martin

Timo Sirainen

16 Nov 16 Nov

1:40 a.m.

On 15.11.2007, at 22.08, Martin R. Ehmsen wrote:

...

The above shows the only time IDLE is every used (except when
Mail.app asks for the capabilities of the server). Somewhere after 14:46:57, but before 14:48:52, I copied (since I
have no SMTP server running on my laptop) a new mail into the new
directory of the maildir I subscribed to. But as you can see from
the above, somehow dovecot failed to notice the new mail. I don't use (d/i)notify or kqueue, but I did set dovecot up to
check every 30 sec, which it obviously didn't. Btw. the mail arrives fine if I manually check for new mail in
Mail.app.

IDLE checks for new mails only in the selected mailbox, not all
mailboxes. IMAP doesn't really support that.

Nikolay Shopik

7:25 a.m.

On 15.11.2007 23:08, Martin R. Ehmsen wrote:

...

On 14/11/2007, at 14.21, Jason Fesler wrote:

...
...
Any hint on how to use tcpdump to only grab the interesting parts?

tcpdump port 143

or whatever.

...
My knowledge of tcpdump is very limited and I only seem to be able to get a lot of noise out of it.

Look at ngrep - it lets you use both tcpdump expressions, *and* further filter by regular expression. And, it shows ascii output instead of all the hex crap - makes it a lot easier to follow ascii based protocols.

Either way, you'll want to disable SSL on the client, so that you can see the traffic properly. :-)

I installed dovecot 1.0.7 on my leopard laptop, disabled SSL and started it on the loopback device. Then I connected to it from Mail.app using IMAP and the IDLE option in Mail.app, while running:

tcpdump -A -s 0 -i lo0 'tcp port imap and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)' > dump.txt

Here is what initially came out of the dump:

<cut a lot of stuff>

14:46:52.589999 IP localhost.50770 > localhost.imap: P 394:404(10) ack 1237 win 65535 E..>..@.@............R...V?2(k.......2..... ...5...57.2 IDLE

14:46:52.590346 IP localhost.imap > localhost.50770: P 1237:1247(10) ack 404 win 65535 E..> .@.@..............R(k...V?<.....2..... ...5...5+ idling

14:46:57.448879 IP localhost.imap > localhost.50777: P 355780982:355781003(21) ack 2104750877 win 65535 E..I..@.@..............Y.4.v}s.......=..... ...f...f* OK Dovecot ready.

14:46:57.459450 IP localhost.50777 > localhost.imap: P 1:17(16) ack 21 win 65535 E..D..@.@............Y..}s...4.......8..... ...f...f1.6 CAPABILITY

14:46:57.459654 IP localhost.imap > localhost.50777: P 21:187(166) ack 17 win 65535 E.....@.@..............Y.4..}s.-........... ...f...f* CAPABILITY IMAP4rev1 SASL-IR SORT THREAD=REFERENCES MULTIAPPEND UNSELECT LITERAL+ IDLE CHILDREN NAMESPACE LOGIN-REFERRALS AUTH=PLAIN 1.6 OK Capability completed.

<and a lot of more stuff and finally it came to a halt>

14:46:57.542907 IP localhost.imap > localhost.50772: P 449:525(76) ack 209 win 65535 E...w.@.@..............T.E....o`.....t..... ...g...g* STATUS "Deleted Messages" (UIDNEXT 3 UNSEEN 0) 7.3 OK Status completed.

14:48:52.590857 IP localhost.imap > localhost.50770: P 1247:1264(17) ack 404 win 65535 E..EI.@.@..............R(k...V?<.....9..... .......5* OK Still here

14:50:52.590661 IP localhost.imap > localhost.50770: P 1264:1281(17) ack 404 win 65535 E..E..@.@..............R(k...V?<.....9..... ........* OK Still here

14:52:52.590964 IP localhost.imap > localhost.50770: P 1281:1298(17) ack 404 win 65535 E..E..@.@..............R(k...V?<.....9..... ........* OK Still here

The above shows the only time IDLE is every used (except when Mail.app asks for the capabilities of the server). Somewhere after 14:46:57, but before 14:48:52, I copied (since I have no SMTP server running on my laptop) a new mail into the new directory of the maildir I subscribed to. But as you can see from the above, somehow dovecot failed to notice the new mail. I don't use (d/i)notify or kqueue, but I did set dovecot up to check every 30 sec, which it obviously didn't. Btw. the mail arrives fine if I manually check for new mail in Mail.app.

I can provide the full dump above if anybody thinks it is going to be useful? Martin

Mail client usually should open few connection to mail server if you have many dirs.

6172

Age (days ago)

6174

Last active (days ago)

List overview

10 comments

5 participants

participants (5)

Benjamin R. Haskell
Jason Fesler
Martin R. Ehmsen
Nikolay Shopik
Timo Sirainen