Re: [Dovecot] Multiple Concurrent IMAP Connections For Same User
Knute Johnson put forth on 3/5/2011 8:36 PM:
On 03/05/2011 12:24 PM, Stan Hoeppner wrote:
Stan Hoeppner put forth on 3/5/2011 1:34 PM: Also, did you delete the sync copies and the GLODA sqlite index file from the user profile? IIRC, TB will still read the files if they exist, even after disabling their parent function. TB just won't write to them any more. Yes, it's stupid, I agree. But I recall running into this problem and needing to delete all the cached mbox files and the sqlite db to get everything working correctly.
Specifically answering your "change" question, the one big difference between TB 3 and TB 2 is GLODA, and there have been a large number of problems with it. I noticed some of them on day one and disabled it.
I use both TB and RoundCube. With both open, I can delete a group of messages in RC in an IMAP subfolder and the messages automatically disappear from my TB view in about 5 seconds, and remain gone. I just performed a test deleting a group of 8 messages via RC in an IMAP subfolder containing list mail from linux-ide. It works fine the other way around as well, though I must manually refresh RC as it's a stateless web app. I just deleted 6 messages in TB, and simply clicking the folder in RC showed the messages were gone. Deleting a message via RC in INBOX required punching the Get Mail button in TB to reveal the message had been deleted. Not sure why the INBOX case required a manual refresh. The messages are all is still gone, after closing and opening TB again.
I don't recall ever having the problem you describe, using TB 3.0 through TB 3.1.9, and Dovecot 1.0.15 through 1.2.15. My point is that TB and Dovecot can be configured properly to eliminate this problem you're having.
Thanks very much for this post as I have been having the same problems with TB and my phone. I did the things you posted but when I delete a message on my phone it doesn't always get deleted in TB. Sometimes it disappears right a way and other times it never goes but it is better than before when it would never delete on TB.
If you're not doing local sync with TB and you have GLODA disabled, TB is going to show you exactly what's in your Dovecot mailbox. If the message doesn't disappear from your TB view sometime after deleting it on your smartphone, then I'd say it's very likely that Dovecot didn't receive the delete and/or expunge command from the phone. Check your Dovecot logs and see what the phone is actually sending, or not sending.
Both of you mentioned problems specifically WRT smartphones. It's likely the IMAP implementation in these phones is simply buggy, or your network coverage is spotty. I don't use a smartphone so I can't really say. What smartphone are you each using?
-- Stan
On 06/03/2011 03:45, Stan Hoeppner wrote:
If you're not doing local sync with TB and you have GLODA disabled, TB is going to show you exactly what's in your Dovecot mailbox. If the message doesn't disappear from your TB view sometime after deleting it on your smartphone, then I'd say it's very likely that Dovecot didn't receive the delete and/or expunge command from the phone. Check your Dovecot logs and see what the phone is actually sending, or not sending.
Both of you mentioned problems specifically WRT smartphones. It's likely the IMAP implementation in these phones is simply buggy, or your network coverage is spotty. I don't use a smartphone so I can't really say. What smartphone are you each using?
Actually it doesn't matter what the second client is (it can be another copy of TB3), the important thing is that TB3 doesn't notice the change underneath it. (The programmers seem to assume that it will have exclusive access to the mailbox from what I can see.) The real mailbox does change correctly, TB3's idea of what the mailbox holds does not and it fails to check its consistency with respect to the server.
Obviously there's a difference in behaviour for some reason when TB3 interacts with Cyrus IMAP as it seems not to have a problem with that IMAP server. It's almost as if TB3 expects to be informed when there has been a deletion. (TB3 will notice other status changes such as a message being read etc.)
The phantom messages within TB3 will also persist between program restarts, which suggests that it's not verifying its cached message list at that point either.
By the way, asking end-users to all change their client settings is not a practical solution. We need to determine why the behaviour between Dovecot and TB3 is different from Cyrus and TB3 and then either ask the TB developers to fix their code or put a work-around in Dovecot. (I may even have to be both.)
Steve
IT Systems Administrator, E-Mail:- steve@earth.ox.ac.uk Department of Earth Sciences, Tel:- +44 (0)1865 282110 Oxford University, South Parks Road, Oxford, UK. Fax:- +44 (0)1865 272072
On 6.3.2011, at 16.51, Stephen Usher wrote:
Obviously there's a difference in behaviour for some reason when TB3 interacts with Cyrus IMAP as it seems not to have a problem with that IMAP server. It's almost as if TB3 expects to be informed when there has been a deletion. (TB3 will notice other status changes such as a message being read etc.)
You could try if setting imap_capabilities to everything except QRESYNC and CONDSTORE would make a difference.
On 3/6/11 10:22 AM, Timo Sirainen wrote:
Obviously there's a difference in behaviour for some reason when TB3 interacts with Cyrus IMAP as it seems not to have a problem with that IMAP server. It's almost as if TB3 expects to be informed when there has been a deletion. (TB3 will notice other status changes such as a message being read etc.)
You could try if setting imap_capabilities to everything except QRESYNC and CONDSTORE would make a difference.
The wiki page I found on this is a little hazy; would this simply be a matter of:
imap_capability = -QRESYNC -CONDSTORE
?
-Dave-- Dave McGuire Port Charlotte, FL
On Sun, 2011-03-06 at 10:47 -0500, Dave McGuire wrote:
On 3/6/11 10:22 AM, Timo Sirainen wrote:
Obviously there's a difference in behaviour for some reason when TB3 interacts with Cyrus IMAP as it seems not to have a problem with that IMAP server. It's almost as if TB3 expects to be informed when there has been a deletion. (TB3 will notice other status changes such as a message being read etc.)
You could try if setting imap_capabilities to everything except QRESYNC and CONDSTORE would make a difference.
The wiki page I found on this is a little hazy; would this simply be a matter of:
imap_capability = -QRESYNC -CONDSTORE
No, there's no -capability feature. You need to list them all:
imap_capability = IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS MULTIAPPEND UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS
And add any plugin capabilities you may have, like QUOTA.
On 03/06/2011 07:22 AM, Timo Sirainen wrote:
On 6.3.2011, at 16.51, Stephen Usher wrote:
Obviously there's a difference in behaviour for some reason when TB3 interacts with Cyrus IMAP as it seems not to have a problem with that IMAP server. It's almost as if TB3 expects to be informed when there has been a deletion. (TB3 will notice other status changes such as a message being read etc.)
You could try if setting imap_capabilities to everything except QRESYNC and CONDSTORE would make a difference.
That stopped dovecot from talking to TB altogether.
--
Knute Johnson knute2011@knutejohnson.com
On 03/06/2011 12:22 PM, Knute Johnson wrote:
On 03/06/2011 07:22 AM, Timo Sirainen wrote:
On 6.3.2011, at 16.51, Stephen Usher wrote:
Obviously there's a difference in behaviour for some reason when TB3 interacts with Cyrus IMAP as it seems not to have a problem with that IMAP server. It's almost as if TB3 expects to be informed when there has been a deletion. (TB3 will notice other status changes such as a message being read etc.)
You could try if setting imap_capabilities to everything except QRESYNC and CONDSTORE would make a difference.
That stopped dovecot from talking to TB altogether.
I take it back. There was a small operator error involved.
That seems to have made it work much better.
Thanks,
knute...
On 06/03/2011 14:51, Stephen Usher wrote:
Actually it doesn't matter what the second client is (it can be another copy of TB3), the important thing is that TB3 doesn't notice the change underneath it. (The programmers seem to assume that it will have exclusive access to the mailbox from what I can see.) The real mailbox does change correctly, TB3's idea of what the mailbox holds does not and it fails to check its consistency with respect to the server.
Obviously there's a difference in behaviour for some reason when TB3 interacts with Cyrus IMAP as it seems not to have a problem with that IMAP server. It's almost as if TB3 expects to be informed when there has been a deletion. (TB3 will notice other status changes such as a message being read etc.)
I may just be having the same issue as you and it's been persistent for a while. I believe that it's possibly a bug related to IDLE in Dovecot (unproven and unsupported claim...)
Do you also get the same effect with read and unread marks?
We have three users here, all logging in from different local machines, which then go through a NAT (so to Dovecot they look the same IP), and all three machines log into the same account with the same credentials (ie shared inbox). We observe that generally the inbox view looks the same on all machines and I haven't noticed any deletes not syncing across instantly, but definitely there exists a set of circumstances where the read/unread marks are not synced correctly. I haven't nailed down the specifics but I believe it might occur when one TB client isn't focused on the shared inbox (background IDLE connection) and it might occur only after some mins have passed in that state (nat timeout?) and it might not occur if CONDSTORE is disabled on TB3...
Do you have a common NAT in your situation? Could there be a NAT timeout/IDLE permutation causing the effect you see?
Note that I can fix the sync issue by simply clicking back into the shared inbox, toggling the state a few times and we are back in again?
My best hunch is that there is some shared code in Dovecot to send IDLE updates every so often and it's hashed on username and IP (I think?), so if we have two users behind nat, one using the connection regularly and the other idle, I wonder if the nat is allowed to timeout on the idle machine? Definitely feels like CONDSTOR is part of the problem, but this would fit because each side of the link would disagree on last update?
Just an idea..
Ed W
On Sun, 2011-03-06 at 20:31 +0000, Ed W wrote:
My best hunch is that there is some shared code in Dovecot to send IDLE updates every so often and it's hashed on username and IP (I think?),
No, only the "OK Still here" notifications are timed like that. If you have inotify/dnotify/kqueue enabled, Dovecot sends the actual mailbox changes to clients immediately when they happen.
On 06/03/2011 22:24, Timo Sirainen wrote:
On Sun, 2011-03-06 at 20:31 +0000, Ed W wrote:
My best hunch is that there is some shared code in Dovecot to send IDLE updates every so often and it's hashed on username and IP (I think?),
No, only the "OK Still here" notifications are timed like that. If you have inotify/dnotify/kqueue enabled, Dovecot sends the actual mailbox changes to clients immediately when they happen.
OK, but following that thought, it still seems possible for three connections behind a NAT to end up with one of them becoming disconnected due to a NAT timeout if the other connections aren't making changes which cause anything to trigger on the IDLE background process? (for ref, router is an Airport Extreme. Believed to have around a 30 min NAT timeout?)
Someone "doing stuff" would cause the "OK Still here" to get backed up for all clients behind the NAT. The NAT will timeout outbound connections individually, so one user keeping their outbound connection alive could cause other connections to die due to backed up "Still here" to that IP?
I guess I could try patching the hash to be less granular - would you be kind enough to remind me where in the code that hash is please?
I'm reasonably sure there is a problem here, just not debugged it enough to capture the problem... It's been a longstanding pain in the arse, just not got serious enough vs the debugging effort that I've tried to tackle it...
Note, in my case I have two machines with TB condstore disabled and they don't show problems, my machine enables condstore and I only believe the problem is on my machine. Hence I think condstore is part of the problem (but that seems consistent with the above theory). Also I only notice the problem when I have not used email for some time, but the other two folks have been... Nat?
Cheers
Ed W
On 3/6/11 6:30 PM, Ed W wrote:
No, only the "OK Still here" notifications are timed like that. If you have inotify/dnotify/kqueue enabled, Dovecot sends the actual mailbox changes to clients immediately when they happen.
OK, but following that thought, it still seems possible for three connections behind a NAT to end up with one of them becoming disconnected due to a NAT timeout if the other connections aren't making changes which cause anything to trigger on the IDLE background process? (for ref, router is an Airport Extreme. Believed to have around a 30 min NAT timeout?)
Someone "doing stuff" would cause the "OK Still here" to get backed up for all clients behind the NAT. The NAT will timeout outbound connections individually, so one user keeping their outbound connection alive could cause other connections to die due to backed up "Still here" to that IP?
I guess I could try patching the hash to be less granular - would you be kind enough to remind me where in the code that hash is please?
I'm reasonably sure there is a problem here, just not debugged it enough to capture the problem... It's been a longstanding pain in the arse, just not got serious enough vs the debugging effort that I've tried to tackle it...
Note, in my case I have two machines with TB condstore disabled and they don't show problems, my machine enables condstore and I only believe the problem is on my machine. Hence I think condstore is part of the problem (but that seems consistent with the above theory). Also I only notice the problem when I have not used email for some time, but the other two folks have been... Nat?
In my case, yes, both clients (iPhone and Thunderbird) are behind NAT from the perspective of the mail server.
That said, I've been testing Timo's imap_capabilities suggestion for about the past 45mins, and it seems to solve the problem for me, at least so far.
-Dave-- Dave McGuire Port Charlotte, FL
On 7.3.2011, at 1.39, Dave McGuire wrote:
That said, I've been testing Timo's imap_capabilities suggestion for about the past 45mins, and it seems to solve the problem for me, at least so far.
So it sounds like there's a bug in someone's QRESYNC code. Either Dovecot or Thunderbird, but could be a bit difficult to find out whose.. A reproducible test case would of course be best.
On 3/6/11 6:46 PM, Timo Sirainen wrote:
That said, I've been testing Timo's imap_capabilities suggestion for about the past 45mins, and it seems to solve the problem for me, at least so far.
So it sounds like there's a bug in someone's QRESYNC code. Either Dovecot or Thunderbird, but could be a bit difficult to find out whose.. A reproducible test case would of course be best.
That'd be tough; it's definitely not repeatable here. It happens maybe 1/3 of the time.
I hate that kind of bug. :-(
-Dave-- Dave McGuire Port Charlotte, FL
On 03/06/2011 03:46 PM, Timo Sirainen wrote:
On 7.3.2011, at 1.39, Dave McGuire wrote:
That said, I've been testing Timo's imap_capabilities suggestion for about the past 45mins, and it seems to solve the problem for me, at least so far.
So it sounds like there's a bug in someone's QRESYNC code. Either Dovecot or Thunderbird, but could be a bit difficult to find out whose.. A reproducible test case would of course be best.
Just a follow up. After adjusting the imap_capability as you suggested I see no left over emails in TB in the long term and only can occasionally get it to not delete one right away. I have turned GLODA back on and that seems to have no effect on the situation at all. I do not sync my messages locally and did not before trying this.
Thanks very much,
--
Knute Johnson knute2011@knutejohnson.com
On 3/8/2011 11:55 AM, Knute Johnson wrote:
On 03/06/2011 03:46 PM, Timo Sirainen wrote:
On 7.3.2011, at 1.39, Dave McGuire wrote:
That said, I've been testing Timo's imap_capabilities suggestion for about the past 45mins, and it seems to solve the problem for me, at least so far.
So it sounds like there's a bug in someone's QRESYNC code. Either Dovecot or Thunderbird, but could be a bit difficult to find out whose.. A reproducible test case would of course be best.
Just a follow up. After adjusting the imap_capability as you suggested I see no left over emails in TB in the long term and only can occasionally get it to not delete one right away. I have turned GLODA back on and that seems to have no effect on the situation at all. I do not sync my messages locally and did not before trying this.
And, shockingly, with the two capabilities removed, TB's notification popup actually reports the number of new messages in the inbox, instead of some random number in the tens of thousands. (dovecot 2.0.11 / TB 3.0.4)
I think you're on to something here.
On 2011-03-08 12:47 PM, Tom Talpey wrote:
(dovecot 2.0.11 / TB 3.0.4)
I think you're on to something here.
You do need to update TB too you know...
The number of bugs squashed since 3.0.4 are too vast too even begin to contemplate...
--
Best regards,
Charles
On 3/8/2011 12:52 PM, Charles Marcus wrote:
On 2011-03-08 12:47 PM, Tom Talpey wrote:
(dovecot 2.0.11 / TB 3.0.4)
I think you're on to something here.
You do need to update TB too you know...
The number of bugs squashed since 3.0.4 are too vast too even begin to contemplate...
It's actually Eudora 1.0, and workarounds are less painful than upgrades. Glad to have this one.
That said, I've been testing Timo's imap_capabilities suggestion for about the past 45mins, and it seems to solve the problem for me, at least so far.
Do you think you could narrow it down to just one of the capabilities being removed? (I think that makes sense?)
Ed W
participants (10)
-
Charles Marcus
-
Dave McGuire
-
Ed W
-
Knute Johnson
-
Knute Johnson
-
Stan Hoeppner
-
Stephen Usher
-
Timo Sirainen
-
Tom Talpey
-
Tom Talpey