[Dovecot] Clients and dovecot-uidlist
Hello,
I'm trying to figure out what exactly (and why) are the consequences of a lost or removal of the dovecot-uidlist file on an IMAP client (Thunderbird for instance).
I took for granted that it was that the client would download all messages again since it might be confused by some UID changes. But I don't really see why (on the IMAP protocol level) and I don't know how to reproduce it.
My understanding is that
. dovecot-uidlist is not recreated until some client selects the mailbox
. the client does something like :
x UID fetch 1:* (FLAGS)
to begin with anyway.So I guess the problem does not occur when no new messages arrive on the mailbox since the same UIDs would be assigned to the same messages ?
But why would that change ? In which situation ?
But can you show me some IMAP protocol command exchange that would illustrate
. when such a problem can occur . how the client would detect that there has been some UID corruption . how it would fix it
Thanks
-- Thomas Hummel
On Tue, Jul 28, 2009 at 07:30:27PM +0200, Thomas Hummel wrote:
I took for granted that it was that the client would download all messages again since it might be confused by some UID changes.
Is it correct ? Timo ?
But I don't really see why (on the IMAP protocol level) and I don't know how to reproduce it.
-- Thomas Hummel | Institut Pasteur <hummel@pasteur.fr> | Pôle informatique - systèmes et réseau
On Thu, Jul 30, 2009 at 07:04:01PM +0200, Thomas Hummel wrote:
On Tue, Jul 28, 2009 at 07:30:27PM +0200, Thomas Hummel wrote:
I took for granted that it was that the client would download all messages again since it might be confused by some UID changes.
Is it correct ? Timo ?
But I don't really see why (on the IMAP protocol level) and I don't know how to reproduce it.
I'm asking this because I'm investigating if CONTROL may be on a local filesystem (for performance reason) instead of an NFS filesystem :
I'm at a point where postfix(+amavisd+clamav) and dovecot on the same machine is too much load. Let's say I've got 2 identical servers and I want to load balance postfix and dovecot between the two. One simple solution would be to :
. run postfix on one server (and having a not started ready to run dovecot on it) . run dovecot on another (and having a not started ready to run postfix on it)
In such a setup, I would want to quickly be able (if the server runing dovecot crashed) to run dovecot on the other. If CONTROL is local, that may mean starting from crash regarding dovecot-uidlist...
Another setup would be to run dovecot and postfix on both servers and to use DNS round-robin, counting on the fact that once a client has done a gethostbyname() or similar, it should have no reaon to change of server (maybe the fact that clients like TB open many connections would cause a problem though).
-- Thomas Hummel | Institut Pasteur <hummel@pasteur.fr> | Pôle informatique - systèmes et réseau
On Thu, 2009-07-30 at 19:24 +0200, Thomas Hummel wrote:
I'm asking this because I'm investigating if CONTROL may be on a local filesystem (for performance reason) instead of an NFS filesystem :
I think it's a bad idea. Besides the UID problems causing all kinds of trouble with clients (some clients ignore UIDVALIDITY completely (Apple Mail, at least used to) and the result is that opening a message from list opens a completely different mail) control files also contain dovecot-keywords which provides the maildir filename letter <-> keyword mapping.
On Thu, Jul 30, 2009 at 01:33:27PM -0400, Timo Sirainen wrote:
I think it's a bad idea. Besides the UID problems causing all kinds of trouble with clients (some clients ignore UIDVALIDITY completely (Apple Mail, at least used to)
Woah! That's indeed stupid from them...
and the result is that opening a message from list opens a completely different mail)
That I don't get since, even if UIDALIDITY change, and even if new messages arrive in the mailbox between dovecot-uidlist erasement and recreation, since INTERNAL DATE is not supposed to change, same messages will get same UIDs won't they ? At least, that's what happens in my tests.
control files also contain dovecot-keywords which provides the maildir filename letter <-> keyword mapping.
Yes, I know about keywords, I didn't mentionned it though.
-- Thomas Hummel | Institut Pasteur <hummel@pasteur.fr> | Pôle informatique - systèmes et réseau
On Jul 31, 2009, at 4:47 AM, Thomas Hummel wrote:
and the result is that opening a message from list opens a completely different mail)
That I don't get since, even if UIDALIDITY change, and even if new
messages arrive in the mailbox between dovecot-uidlist erasement and
recreation, since INTERNAL DATE is not supposed to change, same messages will get same
UIDs won't they ? At least, that's what happens in my tests.
Did you try expunging messages? For example if you have mails with
UIDs 1, 2, 5 and 10 and you delete dovecot-uidlist and dovecot.index*,
Dovecot gives them UIDs 1, 2, 3, 4.
On Fri, Jul 31, 2009 at 11:30:30AM -0400, Timo Sirainen wrote:
For example if you have mails with
UIDs 1, 2, 5 and 10 and you delete dovecot-uidlist and dovecot.index*,
Dovecot gives them UIDs 1, 2, 3, 4.
Ok, I get it.
But the only way I can think of a client can detect that UIDs are not the same for the same messages is by detecting that UIDVALIDITY has changed. So if only dovecot-uidlist gets deleted but index (which store UIDVALIDITY) survives, what will happen in the client ? Will he fetch the wrong data ?
Thanks.
-- Thomas Hummel | Institut Pasteur <hummel@pasteur.fr> | Pôle informatique - systèmes et réseau
On Fri, 2009-07-31 at 18:15 +0200, Thomas Hummel wrote:
On Fri, Jul 31, 2009 at 11:30:30AM -0400, Timo Sirainen wrote:
For example if you have mails with
UIDs 1, 2, 5 and 10 and you delete dovecot-uidlist and dovecot.index*,
Dovecot gives them UIDs 1, 2, 3, 4.Ok, I get it.
But the only way I can think of a client can detect that UIDs are not the same for the same messages is by detecting that UIDVALIDITY has changed. So if only dovecot-uidlist gets deleted but index (which store UIDVALIDITY) survives, what will happen in the client ? Will he fetch the wrong data ?
Well, let's say your uidlist contains:
- uidvalidity = 1234567890
- next_uid = 11
- uid 1 = 1248670280.M981281P9896.host:2,
- uid 5 = 1248673753.M935252P9845.host:2,
- uid 10 = 1248675280.M904006P9872.host:2,
And index contains information:
- uidvalidity = 1234567890
- next_uid = 11
- message uids: 1, 5, 10
If you delete dovecot-uidlist, it's regenerated as:
- uidvalidity = 1234567890
- next_uid = 14
- uid 11 = 1248670280.M981281P9896.host:2,
- uid 12 = 1248673753.M935252P9845.host:2,
- uid 13 = 1248675280.M904006P9872.host:2,
And index is updated:
- uidvalidity = 1234567890
- next_uid = 14
- message uids: 11, 12, 13
i.e. Dovecot gets also the next_uid from index file, and starts assigning UIDs beginning from it.
On Fri, Jul 31, 2009 at 12:26:04PM -0400, Timo Sirainen wrote:
i.e. Dovecot gets also the next_uid from index file, and starts assigning UIDs beginning from it.
Clear. I probably deleted indexes as well in my test and didn't notice a new UIDVALIDITY.
But what about the client's cache ? Does it knows anything about UIDs ?
-- Thomas Hummel | Institut Pasteur <hummel@pasteur.fr> | Pôle informatique - systèmes et réseau
On Fri, 2009-07-31 at 18:40 +0200, Thomas Hummel wrote:
On Fri, Jul 31, 2009 at 12:26:04PM -0400, Timo Sirainen wrote:
i.e. Dovecot gets also the next_uid from index file, and starts assigning UIDs beginning from it.
Clear. I probably deleted indexes as well in my test and didn't notice a new UIDVALIDITY.
But what about the client's cache ? Does it knows anything about UIDs ?
Most clients cache mails based on UIDVALIDITY and UID, so if UIDVALIDITY changes, it re-downloads all mails. If just UIDs change, then it's handled exactly the same way as if all messages just got expunged and added back, so again it'll re-download them.
On Fri, Jul 31, 2009 at 12:49:04PM -0400, Timo Sirainen wrote:
Most clients cache mails based on UIDVALIDITY and UID, so if UIDVALIDITY changes, it re-downloads all mails. If just UIDs change, then it's handled exactly the same way as if all messages just got expunged and added back, so again it'll re-download them.
You mean, one single UID differ between the client's cache and what the server sends and the whole mailbox is fetched again ?
So UIDVALIDITY is just basically the "instance number" of a mailbox, right ?
-- Thomas Hummel | Institut Pasteur <hummel@pasteur.fr> | Pôle informatique - systèmes et réseau
On Fri, 2009-07-31 at 18:56 +0200, Thomas Hummel wrote:
On Fri, Jul 31, 2009 at 12:49:04PM -0400, Timo Sirainen wrote:
Most clients cache mails based on UIDVALIDITY and UID, so if UIDVALIDITY changes, it re-downloads all mails. If just UIDs change, then it's handled exactly the same way as if all messages just got expunged and added back, so again it'll re-download them.
You mean, one single UID differ between the client's cache and what the server sends and the whole mailbox is fetched again ?
No. But if if you delete uidlist, all UIDs change. There are very few situations where only a single UID differs.
Then again, client doesn't know if UID changes. It handles those situations exactly the same way as if message was expunged and a new message was added.
So UIDVALIDITY is just basically the "instance number" of a mailbox, right ?
Pretty much, yes.
On Fri, Jul 31, 2009 at 01:00:10PM -0400, Timo Sirainen wrote:
Then again, client doesn't know if UID changes. It handles those situations exactly the same way as if message was expunged and a new message was added.
So on what criteria does a client fetch a message ? Does it just ask anything above nextUID ?
-- Thomas Hummel | Institut Pasteur <hummel@pasteur.fr> | Pôle informatique - systèmes et réseau
On Fri, 2009-07-31 at 19:15 +0200, Thomas Hummel wrote:
On Fri, Jul 31, 2009 at 01:00:10PM -0400, Timo Sirainen wrote:
Then again, client doesn't know if UID changes. It handles those situations exactly the same way as if message was expunged and a new message was added.
So on what criteria does a client fetch a message ?
Depends on client.
Does it just ask anything above nextUID ?
Outlook just asks for all messages since the last UID it saw.
On Fri, Jul 31, 2009 at 01:18:17PM -0400, Timo Sirainen wrote:
Depends on client.
Ok but I don't understand why you say that a client, though caching UID and UIDVALIDITY, makes no difference between a new message and some UID change on the server ?
-- Thomas Hummel | Institut Pasteur <hummel@pasteur.fr> | Pôle informatique - systèmes et réseau
On Fri, 2009-07-31 at 19:20 +0200, Thomas Hummel wrote:
On Fri, Jul 31, 2009 at 01:18:17PM -0400, Timo Sirainen wrote:
Depends on client.
Ok but I don't understand why you say that a client, though caching UID and UIDVALIDITY, makes no difference between a new message and some UID change on the server ?
Because client thinks UID = message. If a message's UID changes, the client has no idea that it's still the same message.
On Fri, Jul 31, 2009 at 01:22:37PM -0400, Timo Sirainen wrote:
Because client thinks UID = message. If a message's UID changes, the client has no idea that it's still the same message.
Yes but what does and UID has changes from the client pint of view then ? You mean something like, in its cache, the client has a mapping like
slot 1 <-> UID u slot 2 <-> UID v ...
and he sees
slot 1 <-> UID z ?
-- Thomas Hummel | Institut Pasteur <hummel@pasteur.fr> | Pôle informatique - systèmes et réseau
On Fri, 2009-07-31 at 19:27 +0200, Thomas Hummel wrote:
On Fri, Jul 31, 2009 at 01:22:37PM -0400, Timo Sirainen wrote:
Because client thinks UID = message. If a message's UID changes, the client has no idea that it's still the same message.
Yes but what does and UID has changes from the client pint of view then ?
If you have two messages with UIDs 3 and 5, then client makes on difference between:
- UID 3 and 5 are expunged, two new mails are delivered with UIDs 6 and
- Server loses dovecot-uidlist and assigns the existing messages UIDs 6 and 7.
The message contents may be different in those cases, but client doesn't care about that. In both cases it just sees that it used to have UIDs 3 and 5 in its cache, but now server only has UIDs 6 and 7, so it goes and deletes UIDs 3 and 5 from its cache and downloads the messages with UIDs 6 and 7.
You mean something like, in its cache, the client has a mapping like
slot 1 <-> UID u slot 2 <-> UID v ...
and he sees
slot 1 <-> UID z ?
Maybe. Depends on what you mean by slot.
On Fri, Jul 31, 2009 at 01:34:59PM -0400, Timo Sirainen wrote:
Thanks for your explanations and patience Timo!
-- Thomas Hummel | Institut Pasteur <hummel@pasteur.fr> | Pôle informatique - systèmes et réseau
On Fri, Jul 31, 2009 at 12:26:04PM -0400, Timo Sirainen wrote:
So, to come back at my initial question :
Is it correct to say that what's wrong in dovecot-uidlist lost is the potential lost of UIDVALIDITY and nextUID (if indexes are deleted as well) ?
I mean dovecot-uidlist lost alone has not much consequences as long as you still got indexes ?
Is so, sound better than "if you lose dovecot-uidlist, the client downloads everything once again"...
-- Thomas Hummel | Institut Pasteur <hummel@pasteur.fr> | Pôle informatique - systèmes et réseau
On Fri, 2009-07-31 at 18:48 +0200, Thomas Hummel wrote:
On Fri, Jul 31, 2009 at 12:26:04PM -0400, Timo Sirainen wrote:
So, to come back at my initial question :
Is it correct to say that what's wrong in dovecot-uidlist lost is the potential lost of UIDVALIDITY and nextUID (if indexes are deleted as well) ?
Right.
I mean dovecot-uidlist lost alone has not much consequences as long as you still got indexes ?
Clients will still re-download all messages, although they're less likely to break completely. But I still don't understand how you could lose dovecot-uidlist but not indexes.
On Fri, Jul 31, 2009 at 12:50:11PM -0400, Timo Sirainen wrote:
But I still don't understand how you could lose dovecot-uidlist but not indexes.
In a setup like mine where INDEXES and CONTROL point to different places for instance. Is there something wrong with such a solution ?
-- Thomas Hummel | Institut Pasteur <hummel@pasteur.fr> | Pôle informatique - systèmes et réseau
On Fri, 2009-07-31 at 18:59 +0200, Thomas Hummel wrote:
On Fri, Jul 31, 2009 at 12:50:11PM -0400, Timo Sirainen wrote:
But I still don't understand how you could lose dovecot-uidlist but not indexes.
In a setup like mine where INDEXES and CONTROL point to different places for instance. Is there something wrong with such a solution ?
But aren't both of them in local disk then?
On Fri, Jul 31, 2009 at 01:00:38PM -0400, Timo Sirainen wrote:
In a setup like mine where INDEXES and CONTROL point to different places for instance. Is there something wrong with such a solution ?
But aren't both of them in local disk then?
No. At the moment, the Maildirs and the control files are on NFS, indexes are on local disk. That's ok since I've got only one server for now.
Since I plan to split dovecot and postfix on 2 different servers but want to be able to use anyone of them back to run both postfix and dovecot if one of them crash, I plan to put indexes and control files on NFS.
-- Thomas Hummel | Institut Pasteur <hummel@pasteur.fr> | Pôle informatique - systèmes et réseau
On Fri, 2009-07-31 at 19:04 +0200, Thomas Hummel wrote:
On Fri, Jul 31, 2009 at 01:00:38PM -0400, Timo Sirainen wrote:
In a setup like mine where INDEXES and CONTROL point to different places for instance. Is there something wrong with such a solution ?
But aren't both of them in local disk then?
No. At the moment, the Maildirs and the control files are on NFS, indexes are on local disk. That's ok since I've got only one server for now.
Since I plan to split dovecot and postfix on 2 different servers but want to be able to use anyone of them back to run both postfix and dovecot if one of them crash, I plan to put indexes and control files on NFS.
OK. It just sounded like you were thinking about putting index files to NFS and control files to local disk. So there's really no normal situation where only control files could get lost.
On Fri, Jul 31, 2009 at 01:07:30PM -0400, Timo Sirainen wrote:
OK. It just sounded like you were thinking about putting index files to NFS and control files to local disk.
At first I thought about it. But this thread made me understand why it was stupid. Thanks.
-- Thomas Hummel | Institut Pasteur <hummel@pasteur.fr> | Pôle informatique - systèmes et réseau
On Fri, Jul 31, 2009 at 07:04:32PM +0200, Thomas Hummel wrote:
Since I plan to split dovecot and postfix on 2 different servers but want to be able to use anyone of them back to run both postfix and dovecot if one of them crash, I plan to put indexes and control files on NFS.
Another solution would be to DNS round-robin on 2 servers runing dovecot but I think I've read on the dovecot wiki that it wasn't a good idea. Instead, I understood that I could use the IMAP proxy to send the same user on the same server.
However, I wonder if it's not overkill since once the client has resolved the name, the TCP connexion has no reason to change ? But some clients open more than one connexion for one user. That can be a problem I guess...
-- Thomas Hummel | Institut Pasteur <hummel@pasteur.fr> | Pôle informatique - systèmes et réseau
On Fri, 2009-07-31 at 19:08 +0200, Thomas Hummel wrote:
Another solution would be to DNS round-robin on 2 servers runing dovecot but I think I've read on the dovecot wiki that it wasn't a good idea. Instead, I understood that I could use the IMAP proxy to send the same user on the same server.
Right, it'll easily cause all kinds of caching related problems even with the mail_nfs_*=yes settings (which also slow things down).
However, I wonder if it's not overkill since once the client has resolved the name, the TCP connexion has no reason to change ? But some clients open more than one connexion for one user. That can be a problem I guess...
And some users use multiple clients.
BTW. If you're going to run Postfix on both servers and both deliver mails to any users, you're still going to have the same caching problems if you're using Dovecot's deliver.
On Fri, Jul 31, 2009 at 01:15:07PM -0400, Timo Sirainen wrote:
Right, it'll easily cause all kinds of caching related problems even with the mail_nfs_*=yes settings (which also slow things down).
Ok.
And some users use multiple clients.
That's right.
BTW. If you're going to run Postfix on both servers and both deliver mails to any users, you're still going to have the same caching problems if you're using Dovecot's deliver.
No, I'm going to run postfix on one, dovecot on the other (and I don't use deliver for now - I know I should ;-)).
-- Thomas Hummel | Institut Pasteur <hummel@pasteur.fr> | Pôle informatique - systèmes et réseau
On Fri, 2009-07-31 at 19:17 +0200, Thomas Hummel wrote:
BTW. If you're going to run Postfix on both servers and both deliver mails to any users, you're still going to have the same caching problems if you're using Dovecot's deliver.
No, I'm going to run postfix on one, dovecot on the other
Well, that's pretty much the same thing. I meant that if Dovecots on both servers touch the same mailbox at the same time, there are caching problems. Regardless of what part of Dovecot touches it.
(and I don't use deliver for now - I know I should ;-)).
Then there aren't problems, since no dovecot* files are updated.
On Fri, Jul 31, 2009 at 01:20:17PM -0400, Timo Sirainen wrote:
Well, that's pretty much the same thing. I meant that if Dovecots on both servers touch the same mailbox at the same time, there are caching problems. Regardless of what part of Dovecot touches it.
Of course, sorry, I'm not used (yet) to think "deliver".
-- Thomas Hummel | Institut Pasteur <hummel@pasteur.fr> | Pôle informatique - systèmes et réseau
On Tue, 2009-07-28 at 19:30 +0200, Thomas Hummel wrote:
Hello,
I'm trying to figure out what exactly (and why) are the consequences of a lost or removal of the dovecot-uidlist file on an IMAP client (Thunderbird for instance).
Just dovecot-uidlist, or also dovecot.index*? If you delete both, UIDVALIDITY is changed. If you delete only dovecot-uidlist, it'll probably preserve UIDVALIDITY and just give new UIDs to messages (because next_uid is also stored in index file).
I took for granted that it was that the client would download all messages again since it might be confused by some UID changes. But I don't really see why (on the IMAP protocol level) and I don't know how to reproduce it.
My understanding is that
. dovecot-uidlist is not recreated until some client selects the mailbox
. the client does something like :
x UID fetch 1:* (FLAGS) to begin with anyway.So I guess the problem does not occur when no new messages arrive on the mailbox since the same UIDs would be assigned to the same messages ?
Well, Dovecot reads the uidlist file lazily. If the index files still exist and you don't try to open any mail or there are no new mails, then uidlist's existence isn't checked and UIDs won't change.
But can you show me some IMAP protocol command exchange that would illustrate
. when such a problem can occur . how the client would detect that there has been some UID corruption . how it would fix it
Either the client notices that UIDVALIDITY changes on SELECT:
- OK [UIDVALIDITY 1248669921] UIDs valid
Or the UIDs change.
On Thu, Jul 30, 2009 at 01:21:40PM -0400, Timo Sirainen wrote:
Just dovecot-uidlist, or also dovecot.index*?
Just dovecot-uidlist.
If you delete both, UIDVALIDITY is changed. If you delete only dovecot-uidlist, it'll probably preserve UIDVALIDITY and just give new UIDs to messages (because next_uid is also stored in index file).
I don't get that : do indexes store UIDVALIDITY ? nextUID isn't enough to guaranty mailbox instance uniqueness, right ?
And, same as in my previous answer : why would they get new UIDs ? As they will be "sorted" (maybe because of INTERNAL DATE) in the same order ? When I erase dovecot-uidlist manually, it gets recreated with the same UIDs (but I didn't delete indexes though)...
Well, Dovecot reads the uidlist file lazily. If the index files still exist and you don't try to open any mail or there are no new mails, then uidlist's existence isn't checked and UIDs won't change.
Same as above : I cannot get them to change in a test situation where I delete dovecot-uidlist, send a new mail and check mail again.
Thanks
-- Thomas Hummel | Institut Pasteur <hummel@pasteur.fr> | Pôle informatique - systèmes et réseau
On Jul 31, 2009, at 5:25 AM, Thomas Hummel wrote:
On Thu, Jul 30, 2009 at 01:21:40PM -0400, Timo Sirainen wrote:
Just dovecot-uidlist, or also dovecot.index*?
Just dovecot-uidlist.
Then I don't really get it. You want to store index files on NFS, but
not control files? Doesn't really make sense to me.
If you delete both, UIDVALIDITY is changed. If you delete only dovecot-uidlist, it'll probably preserve UIDVALIDITY and just give
new UIDs to messages (because next_uid is also stored in index file).I don't get that : do indexes store UIDVALIDITY ?
Yes.
nextUID isn't enough to guaranty mailbox instance uniqueness, right ?
Right.
BTW. You can use idxview program to look at what index file's contents
are.
participants (2)
-
Thomas Hummel
-
Timo Sirainen