Re: Outlook 2010 woes
Old server:
- Ubuntu 10.04.4 LTS
- Dovecot 2.1.13
- Maildir++
- Local auth via passwd/shadow files
New server:
- Debian GNU/Linux 8.6
- Dovecot 2.2.13
- Maildir++
- Quotas enabled
- LDAP
Basically what's happening is that users are seeing large delays when navigating between different IMAP folders. So, for example, user "X" is sitting idle in their INBOX.
Rebuilding caches? Do you get the same delay when going back to the folder after the initial delay.
Joseph Tam jtam.home@gmail.com
On 10/12/16 4:11 PM, Joseph Tam wrote:
Old server:
- Ubuntu 10.04.4 LTS
- Dovecot 2.1.13
- Maildir++
- Local auth via passwd/shadow files
New server:
- Debian GNU/Linux 8.6
- Dovecot 2.2.13
- Maildir++
- Quotas enabled
- LDAP
Basically what's happening is that users are seeing large delays when navigating between different IMAP folders. So, for example, user "X" is sitting idle in their INBOX.
Rebuilding caches? Do you get the same delay when going back to the folder after the initial delay.
Joseph Tam jtam.home@gmail.com
No, but once sitting idle again for 10-15 seconds, the delay occurs again regardless of which folder you choose.
Am I understanding your question correctly? It really seems to me like Outlook is prematurely ending IMAP sessions.
I also extended the "Server Timeout" setting in OT2010 to 10 minutes, which doesn't seem to help either. (!)
I was considering enabling the auth_cache feature to see if that helps.
I'll let the list know what happens -- planning on doing that today.
On Thu, 13 Oct 2016 08:36:23 -0500, Bryan Holloway stated:
I also extended the "Server Timeout" setting in OT2010 to 10 minutes, which doesn't seem to help either. (!)
Outlook 2010 is a very old version. Why not update to the 2016 version. I am running it without any problems. If you do update, remember to remove the old version completely first.
-- Jerry
On 10/13/16 8:55 AM, Jerry wrote:
On Thu, 13 Oct 2016 08:36:23 -0500, Bryan Holloway stated:
I also extended the "Server Timeout" setting in OT2010 to 10 minutes, which doesn't seem to help either. (!)
Outlook 2010 is a very old version. Why not update to the 2016 version. I am running it without any problems. If you do update, remember to remove the old version completely first.
Yeah -- totally not disagreeing with that statement ... the problem is that the customer is putting their foot down since everything worked fine with Dovecot 2.1.
But yes, I have mentioned that to them ...
On 10/13/16 9:06 AM, Bryan Holloway wrote:
On 10/13/16 8:55 AM, Jerry wrote:
On Thu, 13 Oct 2016 08:36:23 -0500, Bryan Holloway stated:
I also extended the "Server Timeout" setting in OT2010 to 10 minutes, which doesn't seem to help either. (!)
Outlook 2010 is a very old version. Why not update to the 2016 version. I am running it without any problems. If you do update, remember to remove the old version completely first.
Yeah -- totally not disagreeing with that statement ... the problem is that the customer is putting their foot down since everything worked fine with Dovecot 2.1.
But yes, I have mentioned that to them ...
I guess I should add that it would be one thing if there were a specific IMAP feature that a newer Dovecot version (2.2) supported and the client didn't, but I haven't been able to pinpoint it.
Obviously the behavior is different than what it was, but it would be a lot easier to convince the customer to upgrade if I could point a finger right at the "feature" in question.
In the meantime, I have to try and figure out what's changed ...
On October 13, 2016 at 4:55 PM Jerry jerry@seibercom.net wrote:
On Thu, 13 Oct 2016 08:36:23 -0500, Bryan Holloway stated:
I also extended the "Server Timeout" setting in OT2010 to 10 minutes, which doesn't seem to help either. (!)
Outlook 2010 is a very old version. Why not update to the 2016 version. I am running it without any problems. If you do update, remember to remove the old version completely first.
-- Jerry
I do wonder if the real culprit is some firewall that timeouts the idle connection.
Aki
On 10/13/16 9:07 AM, Aki Tuomi wrote:
On October 13, 2016 at 4:55 PM Jerry jerry@seibercom.net wrote:
On Thu, 13 Oct 2016 08:36:23 -0500, Bryan Holloway stated:
I also extended the "Server Timeout" setting in OT2010 to 10 minutes, which doesn't seem to help either. (!)
Outlook 2010 is a very old version. Why not update to the 2016 version. I am running it without any problems. If you do update, remember to remove the old version completely first.
-- Jerry
I do wonder if the real culprit is some firewall that timeouts the idle connection.
Aki
I considered that, but again everything worked fine until we moved them from 2.1 to 2.2. Their same firewall is in use.
Is there a way to see the IMAP commands coming from the client? I've tried looking at PCAPs, but of course they're encrypted so I can't see the actual dialog going on between the server and client. I didn't see an obvious way to do this in the docs.
On Thu, 13 Oct 2016 09:53:19 -0500 Bryan Holloway bryan@shout.net wrote:
[...]
Is there a way to see the IMAP commands coming from the client? I've tried looking at PCAPs, but of course they're encrypted so I can't see the actual dialog going on between the server and client. I didn't see an obvious way to do this in the docs.
If you have access to the SSL/TLS key (IOW, the private part of the
cert) the server uses to secure IMAP connections you can dump the IMAP
traffic using the ssldump utility (which builds on tcpdump).
On 10/13/16 10:23 AM, Konstantin Khomoutov wrote:
On Thu, 13 Oct 2016 09:53:19 -0500 Bryan Holloway bryan@shout.net wrote:
[...]
Is there a way to see the IMAP commands coming from the client? I've tried looking at PCAPs, but of course they're encrypted so I can't see the actual dialog going on between the server and client. I didn't see an obvious way to do this in the docs.
If you have access to the SSL/TLS key (IOW, the private part of the cert) the server uses to secure IMAP connections you can dump the IMAP traffic using the
ssldumputility (which builds ontcpdump).
I do, but the client is using a DH key exchange so I only have the server-side private key.
Tried that using Wireshark's decoder features and ran into this problem. I'm assuming I'd run into the same using ssldump, but I'll give it a shot!
Stupid privacy. :)
On Thu, 13 Oct 2016 10:35:14 -0500 Bryan Holloway bryan@shout.net wrote:
[...]
Is there a way to see the IMAP commands coming from the client? I've tried looking at PCAPs, but of course they're encrypted so I can't see the actual dialog going on between the server and client. I didn't see an obvious way to do this in the docs.
If you have access to the SSL/TLS key (IOW, the private part of the cert) the server uses to secure IMAP connections you can dump the IMAP traffic using the
ssldumputility (which builds ontcpdump).I do, but the client is using a DH key exchange so I only have the server-side private key.
Tried that using Wireshark's decoder features and ran into this problem. I'm assuming I'd run into the same using ssldump, but I'll give it a shot!
I think DH is not the culprit: just to be able to actually decode SSL traffic, you must have the server private key when you're decoding the SSL handshake phase -- to be able to recover the session keys, which you then use to decode the actual tunneled data.
On October 13, 2016 at 6:52 PM Konstantin Khomoutov flatworm@users.sourceforge.net wrote:
On Thu, 13 Oct 2016 10:35:14 -0500 Bryan Holloway bryan@shout.net wrote:
[...]
Is there a way to see the IMAP commands coming from the client? I've tried looking at PCAPs, but of course they're encrypted so I can't see the actual dialog going on between the server and client. I didn't see an obvious way to do this in the docs.
If you have access to the SSL/TLS key (IOW, the private part of the cert) the server uses to secure IMAP connections you can dump the IMAP traffic using the
ssldumputility (which builds ontcpdump).I do, but the client is using a DH key exchange so I only have the server-side private key.
Tried that using Wireshark's decoder features and ran into this problem. I'm assuming I'd run into the same using ssldump, but I'll give it a shot!
I think DH is not the culprit: just to be able to actually decode SSL traffic, you must have the server private key when you're decoding the SSL handshake phase -- to be able to recover the session keys, which you then use to decode the actual tunneled data.
You can also enable only non DH algorithms in ssl settings if rawlog isn't working for you.
Aki
On 10/13/16 11:01 AM, Aki Tuomi wrote:
On October 13, 2016 at 6:52 PM Konstantin Khomoutov flatworm@users.sourceforge.net wrote:
On Thu, 13 Oct 2016 10:35:14 -0500 Bryan Holloway bryan@shout.net wrote:
[...]
Is there a way to see the IMAP commands coming from the client? I've tried looking at PCAPs, but of course they're encrypted so I can't see the actual dialog going on between the server and client. I didn't see an obvious way to do this in the docs.
If you have access to the SSL/TLS key (IOW, the private part of the cert) the server uses to secure IMAP connections you can dump the IMAP traffic using the
ssldumputility (which builds ontcpdump).I do, but the client is using a DH key exchange so I only have the server-side private key.
Tried that using Wireshark's decoder features and ran into this problem. I'm assuming I'd run into the same using ssldump, but I'll give it a shot!
I think DH is not the culprit: just to be able to actually decode SSL traffic, you must have the server private key when you're decoding the SSL handshake phase -- to be able to recover the session keys, which you then use to decode the actual tunneled data.
You can also enable only non DH algorithms in ssl settings if rawlog isn't working for you.
Aki
Ah -- interesting tip. I hadn't thought of that. Thank you! I'll report my findings to the list.
So after several days of more troubleshooting, I have some things to report to the list.
First and foremost, I have discovered that the issue has nothing to do with SSL/TLS, which was my earlier suspicion because after doing some PCAPs I discovered that the transactions were negotiating TLS 1.2 on the new server, as opposed to 1.0 on the old.
Also thank you for the rawlog suggestion: that helped a lot in determining what was happening on the IMAP level.
That all said, this is what I've discovered:
There is a very curious and reproducible four-second delay during the negotiation between server and client which is not present in Dovecot 2.1. This is what our customer is complaining about using Outlook 2010.
During a plaintext TCP stream, I'm seeing this:
Client connects (SYN) to server.
Server ACKs and throws back CAPABILITIES.
User attempts to auth with DIGEST-MD5.
Server says, "no thanks." (Not sure why, but I don't believe this is relevant.)
User attempts to auth with plaintext.
Server says, "Yup. You are you. You're logged in."
Client sends the following: ID ("name" "Microsoft Outlook" "version" "14.0")
Server sends an ACK
... and then there's this very curious four-second delay.
- Server then sends out new CAPABILITIES, and everything proceeds thereafter as normal and zippy and fast.
Does this shed any light on the subject?
On 10/13/16 11:21 AM, Bryan Holloway wrote:
On 10/13/16 11:01 AM, Aki Tuomi wrote:
On October 13, 2016 at 6:52 PM Konstantin Khomoutov flatworm@users.sourceforge.net wrote:
On Thu, 13 Oct 2016 10:35:14 -0500 Bryan Holloway bryan@shout.net wrote:
[...]
Is there a way to see the IMAP commands coming from the client? I've tried looking at PCAPs, but of course they're encrypted so I can't see the actual dialog going on between the server and client. I didn't see an obvious way to do this in the docs.
If you have access to the SSL/TLS key (IOW, the private part of the cert) the server uses to secure IMAP connections you can dump the IMAP traffic using the
ssldumputility (which builds ontcpdump).I do, but the client is using a DH key exchange so I only have the server-side private key.
Tried that using Wireshark's decoder features and ran into this problem. I'm assuming I'd run into the same using ssldump, but I'll give it a shot!
I think DH is not the culprit: just to be able to actually decode SSL traffic, you must have the server private key when you're decoding the SSL handshake phase -- to be able to recover the session keys, which you then use to decode the actual tunneled data.
You can also enable only non DH algorithms in ssl settings if rawlog isn't working for you.
Aki
Ah -- interesting tip. I hadn't thought of that. Thank you! I'll report my findings to the list.
In case anyone is interested, we finally found the problem:
The new (2.2) server had "auth_mechanisms" of "digest-md5" enabled along with "plain". This is what was causing the four-second delay, but only with Outlook clients.
Everything is working great now across the board.
Thanks again to everyone's suggestions.
- bryanOn 10/27/16 7:09 PM, Bryan Holloway wrote:
So after several days of more troubleshooting, I have some things to report to the list.
First and foremost, I have discovered that the issue has nothing to do with SSL/TLS, which was my earlier suspicion because after doing some PCAPs I discovered that the transactions were negotiating TLS 1.2 on the new server, as opposed to 1.0 on the old.
Also thank you for the rawlog suggestion: that helped a lot in determining what was happening on the IMAP level.
That all said, this is what I've discovered:
There is a very curious and reproducible four-second delay during the negotiation between server and client which is not present in Dovecot 2.1. This is what our customer is complaining about using Outlook 2010.
During a plaintext TCP stream, I'm seeing this:
Client connects (SYN) to server.
Server ACKs and throws back CAPABILITIES.
User attempts to auth with DIGEST-MD5.
Server says, "no thanks." (Not sure why, but I don't believe this is relevant.)
User attempts to auth with plaintext.
Server says, "Yup. You are you. You're logged in."
Client sends the following: ID ("name" "Microsoft Outlook" "version" "14.0")
Server sends an ACK
... and then there's this very curious four-second delay.
- Server then sends out new CAPABILITIES, and everything proceeds thereafter as normal and zippy and fast.
Does this shed any light on the subject?
On 10/13/16 11:21 AM, Bryan Holloway wrote:
On 10/13/16 11:01 AM, Aki Tuomi wrote:
On October 13, 2016 at 6:52 PM Konstantin Khomoutov flatworm@users.sourceforge.net wrote:
On Thu, 13 Oct 2016 10:35:14 -0500 Bryan Holloway bryan@shout.net wrote:
[...] > Is there a way to see the IMAP commands coming from the client? > I've tried looking at PCAPs, but of course they're encrypted so I > can't see the actual dialog going on between the server and > client. I didn't see an obvious way to do this in the docs.
If you have access to the SSL/TLS key (IOW, the private part of the cert) the server uses to secure IMAP connections you can dump the IMAP traffic using the
ssldumputility (which builds ontcpdump).I do, but the client is using a DH key exchange so I only have the server-side private key.
Tried that using Wireshark's decoder features and ran into this problem. I'm assuming I'd run into the same using ssldump, but I'll give it a shot!
I think DH is not the culprit: just to be able to actually decode SSL traffic, you must have the server private key when you're decoding the SSL handshake phase -- to be able to recover the session keys, which you then use to decode the actual tunneled data.
You can also enable only non DH algorithms in ssl settings if rawlog isn't working for you.
Aki
Ah -- interesting tip. I hadn't thought of that. Thank you! I'll report my findings to the list.
On 11/1/16 6:20 PM, Bryan Holloway wrote:
In case anyone is interested, we finally found the problem:
The new (2.2) server had "auth_mechanisms" of "digest-md5" enabled along with "plain". This is what was causing the four-second delay, but only with Outlook clients.
Everything is working great now across the board.
Thanks again to everyone's suggestions.
- bryan
Sorry -- that wasn't very clear, was it:
Removing "digest-md5" fixed the issue.
On 10/27/16 7:09 PM, Bryan Holloway wrote:
So after several days of more troubleshooting, I have some things to report to the list.
First and foremost, I have discovered that the issue has nothing to do with SSL/TLS, which was my earlier suspicion because after doing some PCAPs I discovered that the transactions were negotiating TLS 1.2 on the new server, as opposed to 1.0 on the old.
Also thank you for the rawlog suggestion: that helped a lot in determining what was happening on the IMAP level.
That all said, this is what I've discovered:
There is a very curious and reproducible four-second delay during the negotiation between server and client which is not present in Dovecot 2.1. This is what our customer is complaining about using Outlook 2010.
During a plaintext TCP stream, I'm seeing this:
Client connects (SYN) to server.
Server ACKs and throws back CAPABILITIES.
User attempts to auth with DIGEST-MD5.
Server says, "no thanks." (Not sure why, but I don't believe this is relevant.)
User attempts to auth with plaintext.
Server says, "Yup. You are you. You're logged in."
Client sends the following: ID ("name" "Microsoft Outlook" "version" "14.0")
Server sends an ACK
... and then there's this very curious four-second delay.
- Server then sends out new CAPABILITIES, and everything proceeds thereafter as normal and zippy and fast.
Does this shed any light on the subject?
On 10/13/16 11:21 AM, Bryan Holloway wrote:
On 10/13/16 11:01 AM, Aki Tuomi wrote:
On October 13, 2016 at 6:52 PM Konstantin Khomoutov flatworm@users.sourceforge.net wrote:
On Thu, 13 Oct 2016 10:35:14 -0500 Bryan Holloway bryan@shout.net wrote:
> [...] >> Is there a way to see the IMAP commands coming from the client? >> I've tried looking at PCAPs, but of course they're encrypted so I >> can't see the actual dialog going on between the server and >> client. I didn't see an obvious way to do this in the docs. > > If you have access to the SSL/TLS key (IOW, the private part of the > cert) the server uses to secure IMAP connections you can dump the > IMAP traffic using the
ssldumputility (which builds on >tcpdump).I do, but the client is using a DH key exchange so I only have the server-side private key.
Tried that using Wireshark's decoder features and ran into this problem. I'm assuming I'd run into the same using ssldump, but I'll give it a shot!
I think DH is not the culprit: just to be able to actually decode SSL traffic, you must have the server private key when you're decoding the SSL handshake phase -- to be able to recover the session keys, which you then use to decode the actual tunneled data.
You can also enable only non DH algorithms in ssl settings if rawlog isn't working for you.
Aki
Ah -- interesting tip. I hadn't thought of that. Thank you! I'll report my findings to the list.
On Tue, 1 Nov 2016 18:20:14 -0500 Bryan Holloway bryan@shout.net wrote:
In case anyone is interested, we finally found the problem:
The new (2.2) server had "auth_mechanisms" of "digest-md5" enabled along with "plain". This is what was causing the four-second delay, but only with Outlook clients.
Everything is working great now across the board.
Thanks again to everyone's suggestions.
Thanks for sharing.
It's pretty amazing how far removed the problem cause can be from that problem's manifestation ;-)
Am 2016-11-02 um 13:16 schrieb Konstantin Khomoutov:
On Tue, 1 Nov 2016 18:20:14 -0500 Bryan Holloway bryan@shout.net wrote:
The new (2.2) server had "auth_mechanisms" of "digest-md5" enabled along with "plain". This is what was causing the four-second delay, but only with Outlook clients.
It's pretty amazing how far removed the problem cause can be from that problem's manifestation ;-)
Nah, this is not far at all; a system with virtual users only and PAM configured will take even longer to authenticate than four seconds
PS: I should have posted earlier ;)
-- peter
Am 13.10.2016 um 16:53 schrieb Bryan Holloway:
On 10/13/16 9:07 AM, Aki Tuomi wrote:
On October 13, 2016 at 4:55 PM Jerry jerry@seibercom.net wrote:
On Thu, 13 Oct 2016 08:36:23 -0500, Bryan Holloway stated:
I also extended the "Server Timeout" setting in OT2010 to 10 minutes, which doesn't seem to help either. (!)
Outlook 2010 is a very old version. Why not update to the 2016 version. I am running it without any problems. If you do update, remember to remove the old version completely first.
-- Jerry
I do wonder if the real culprit is some firewall that timeouts the idle connection.
Aki
I considered that, but again everything worked fine until we moved them from 2.1 to 2.2. Their same firewall is in use.
Is there a way to see the IMAP commands coming from the client? I've tried looking at PCAPs, but of course they're encrypted so I can't see the actual dialog going on between the server and client. I didn't see an obvious way to do this in the docs.
There is a "rawlog" feature, which writes down the hole decrypted imap session in files.
... service imap { ... executable = imap postlogin ... }
...
service postlogin { executable = script-login -d rawlog unix_listener postlogin { } } ...
This should write *.in an *.out files to "$mail_location/dovecot.rawlog/" directory for each imap session. The directory should be writeable by the dovecot user. I tested this some years ago, so I'm not shure if the configuration is still valid.
Regards Urban
On 10/13/16 10:42 AM, Urban Loesch wrote:
Am 13.10.2016 um 16:53 schrieb Bryan Holloway:
On 10/13/16 9:07 AM, Aki Tuomi wrote:
On October 13, 2016 at 4:55 PM Jerry jerry@seibercom.net wrote:
On Thu, 13 Oct 2016 08:36:23 -0500, Bryan Holloway stated:
I also extended the "Server Timeout" setting in OT2010 to 10 minutes, which doesn't seem to help either. (!)
Outlook 2010 is a very old version. Why not update to the 2016 version. I am running it without any problems. If you do update, remember to remove the old version completely first.
-- Jerry
I do wonder if the real culprit is some firewall that timeouts the idle connection.
Aki
I considered that, but again everything worked fine until we moved them from 2.1 to 2.2. Their same firewall is in use.
Is there a way to see the IMAP commands coming from the client? I've tried looking at PCAPs, but of course they're encrypted so I can't see the actual dialog going on between the server and client. I didn't see an obvious way to do this in the docs.
There is a "rawlog" feature, which writes down the hole decrypted imap session in files.
... service imap { ... executable = imap postlogin ... }
...
service postlogin { executable = script-login -d rawlog unix_listener postlogin { } } ...
This should write *.in an *.out files to "$mail_location/dovecot.rawlog/" directory for each imap session. The directory should be writeable by the dovecot user. I tested this some years ago, so I'm not shure if the configuration is still valid.
Regards Urban
Great! I will try this.
On Thu, 13 Oct 2016, Bryan Holloway wrote:
Rebuilding caches? Do you get the same delay when going back to the folder after the initial delay.
No, but once sitting idle again for 10-15 seconds, the delay occurs again regardless of which folder you choose.
Another diagnostic is to strace the server process.
Joseph Tam jtam.home@gmail.com
participants (7)
-
Aki Tuomi
-
Bryan Holloway
-
Jerry
-
Joseph Tam
-
Konstantin Khomoutov
-
Peter Chiochetti
-
Urban Loesch