[Dovecot] Why is "UIDVALIDITY changed" fatal?
I've got several users testing Dovecot IMAP at the moment, but some of them are also accessing their mail via UW-IMAP at other times. (I've warned them about accessing the same folder simultaneously via UW-IMAP and Dovecot!) In addition, we have special folders "AutoSpam" and "Trash" that pruned using Washington's "mailutil" automatically every so often.
Of course, Dovecot's indexes will become invalid in such cases, but they seem to get rebuilt automatically OK.
However, in some cases UW-IMAP seems to change the UIDVALIDITY value (not sure why!) and when Dovecot tries to sync it notices this and gives up with a fatal error on opening that folder. Opening the folder a second time is then OK.
I was wondering why Dovecot can't just rebuild the indexes silently in such cases (which seems to be done by returning "0" from the mbox_sync_loop() routine)?
Best Wishes, Chris
--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-- Christopher Wakelin, c.d.wakelin@reading.ac.uk IT Services Centre, The University of Reading, Tel: +44 (0)118 378 8439 Whiteknights, Reading, RG6 2AF, UK Fax: +44 (0)118 975 3094
On Wed, 20 Apr 2005, Chris Wakelin wrote:
I've got several users testing Dovecot IMAP at the moment, but some of them are also accessing their mail via UW-IMAP at other times. (I've warned them about accessing the same folder simultaneously via UW-IMAP and Dovecot!) In addition, we have special folders "AutoSpam"
However, in some cases UW-IMAP seems to change the UIDVALIDITY value (not sure why!) and when Dovecot tries to sync it notices this and gives up with a fatal error on opening that folder. Opening the folder a second time is then OK.
I have this problem as well, but I don't think that it is UW doing it in my case. If I switch my mailbox back and forth, I don't see the problem.
The most common case here is some of our support mailboxes where several people log into them at once (I know, they shouldn't...). This causes all sorts of these errors, but dovecot recovers from them and the support staff doesn't seem to notice. Also, the mbox hasn't become corrupted so far, which is a really good thing.
Maybe your users are doing the same thing?
Todd
Todd Burroughs wrote:
I have this problem as well, but I don't think that it is UW doing it in my case. If I switch my mailbox back and forth, I don't see the problem.
I think it happens when a large number of messages are added or removed. E.g. the prune script I mentioned will delete all messages older than a week, which could be a lot in the case of my Spam folder ;-)
It's not clear to me (yet) why UW-IMAP needs to change UIDVALIDITY; it's usually just the output of time(0) and I don't think it should be running out of UIDs (a 32-bit number!) for the messages. My guess is the X-IMAPbase header, where UIDVALIDITY is stored, is disappearing somehow.
The most common case here is some of our support mailboxes where several people log into them at once (I know, they shouldn't...). This causes all sorts of these errors, but dovecot recovers from them and the support staff doesn't seem to notice. Also, the mbox hasn't become corrupted so far, which is a really good thing.
Maybe your users are doing the same thing?
Don't think so. I've been testing multiple accesses to my INBOX with only transient problems due to clients caching stuff. I don't think my colleagues have been doing so, though.
I should also say it's mboxes, Dovecot-1.0-stable, and local disk (maildirs over NFS are in the future, possibly!).
Best Wishes, Chris
-- --+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+- Christopher Wakelin, c.d.wakelin@reading.ac.uk IT Services Centre, The University of Reading, Tel: +44 (0)118 378 8439 Whiteknights, Reading, RG6 2AF, UK Fax: +44 (0)118 975 3094
On 20.4.2005, at 21:01, Chris Wakelin wrote:
However, in some cases UW-IMAP seems to change the UIDVALIDITY value (not sure why!) and when Dovecot tries to sync it notices this and gives up with a fatal error on opening that folder. Opening the folder a second time is then OK.
When all messages are expunged, UW-IMAP and newer Dovecots (2005-02-08 in -stable) should write a "pseudo mail" (Subject: DON'T DELETE THIS MESSAGE..) to make sure the UIDVALIDITY isn't lost. Doesn't this always happen with you?
I was wondering why Dovecot can't just rebuild the indexes silently in such cases
The easiest way to handle it was to just mark the index corrupted and let normal error handling paths deal with it. I changed it now in CVS not to do this.
(which seems to be done by returning "0" from the mbox_sync_loop() routine)?
It re-syncs the whole mailbox, but it doesn't recreate the index files after they've been marked as corrupted.
Timo Sirainen wrote:
When all messages are expunged, UW-IMAP and newer Dovecots (2005-02-08 in -stable) should write a "pseudo mail" (Subject: DON'T DELETE THIS MESSAGE..) to make sure the UIDVALIDITY isn't lost. Doesn't this always happen with you?
I'm not sure quite why it's happening. I haven't been able to reproduce it on demand yet. The new UIDVALIDITY seem to indicate it's changing as part of the mailutil command (at about 7:15am - UW-IMAP uses time(0) as the value).
I've been using 1.0-stable from yesterday (2005-04-22) and it didn't happen to my AutoSpam folder this morning.
I let you know if I find the cause!
Chris
-- --+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+- Christopher Wakelin, c.d.wakelin@reading.ac.uk IT Services Centre, The University of Reading, Tel: +44 (0)118 378 8439 Whiteknights, Reading, RG6 2AF, UK Fax: +44 (0)118 975 3094
Aha! I've managed to reproduce the UIDVALIDITY problem using dovecot-stable-20050411. Sure enough, it appears to be simply that dovecot hasn't updated the UID-NEXT field, so UW-IMAP (actually mailutil) spots larger UIDs, complains, and resets the UIDVALIDITY.
So the fix in dovecot-stable-20050422 should work (which in my limited testing, it does!).
Now, we just need to find out why it's syncing so many messages when it reads a new one. Let me know if there's any more i_info lines I could stick in to find out what's going on.
Does uid=0 (i.e. "sync_loop 0") mean something special? When it was doing all that logging it looks like it sometimes went through the whole mailbox twice. I had a look at the mbox-sync.c file and soon got lost :(
Best Wishes, Chris
Chris Wakelin wrote:
I'm not sure quite why it's happening. I haven't been able to reproduce it on demand yet. The new UIDVALIDITY seem to indicate it's changing as part of the mailutil command (at about 7:15am - UW-IMAP uses time(0) as the value).
I've been using 1.0-stable from yesterday (2005-04-22) and it didn't happen to my AutoSpam folder this morning.
I let you know if I find the cause!
Chris
-- --+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+- Christopher Wakelin, c.d.wakelin@reading.ac.uk IT Services Centre, The University of Reading, Tel: +44 (0)118 378 8439 Whiteknights, Reading, RG6 2AF, UK Fax: +44 (0)118 975 3094
Now, we just need to find out why it's syncing so many messages when it reads a new one. Let me know if there's any more i_info lines I could stick in to find out what's going on.
Does uid=0 (i.e. "sync_loop 0") mean something special? When it was doing all that logging it looks like it sometimes went through the whole mailbox twice. I had a look at the mbox-sync.c file and soon got lost :(
Almost sent you a debugging patch, but then just before sending it I thought I'd try if it matters if the first message is the "folder internal data" mail. That's the problem.
New tarballs should be ready within a few minutes.
Timo Sirainen wrote:
Almost sent you a debugging patch, but then just before sending it I thought I'd try if it matters if the first message is the "folder internal data" mail. That's the problem.
Yep, certainly my INBOX has the pseudo-message - I've had it a while ;) The AutoSpam folder probably won't have, though (UW-IMAP doesn't add it if it's missing, but does if it creates the folder, I think).
New tarballs should be ready within a few minutes.
I've compiled it up, and looks good so far :)
Best Wishes, Chris
-- --+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+- Christopher Wakelin, c.d.wakelin@reading.ac.uk IT Services Centre, The University of Reading, Tel: +44 (0)118 378 8439 Whiteknights, Reading, RG6 2AF, UK Fax: +44 (0)118 975 3094
Hmm, now I'm seeing problems with flag updates. It seems dovecot isn't always adding X-Status and X-UID headers (or updating the latter) when a message is read - I seem to have a mixture of "Status: O" or both headers missing. Is this because of mbox_dirty_syncs?
It didn't used to happen before today's fixes (but did happen before the very latest ones) Symptoms are Thunderbird (or Pine) think they've updated the flags, but when the folder is reopened, it reverts. I've tried deleting the dovecot.index* files.
Best Wishes, Chris
Chris Wakelin wrote:
Timo Sirainen wrote:
Almost sent you a debugging patch, but then just before sending it I thought I'd try if it matters if the first message is the "folder internal data" mail. That's the problem.
Yep, certainly my INBOX has the pseudo-message - I've had it a while ;) The AutoSpam folder probably won't have, though (UW-IMAP doesn't add it if it's missing, but does if it creates the folder, I think).
New tarballs should be ready within a few minutes.
I've compiled it up, and looks good so far :)
Best Wishes, Chris
-- --+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+- Christopher Wakelin, c.d.wakelin@reading.ac.uk IT Services Centre, The University of Reading, Tel: +44 (0)118 378 8439 Whiteknights, Reading, RG6 2AF, UK Fax: +44 (0)118 975 3094
I just looked in the logs (sorry, should have done that first) and it's been giving errors "mbox sync: UID inserted in the middle of mailbox .../INBOX (xxx > yyy)".
Chris
Chris Wakelin wrote:
Hmm, now I'm seeing problems with flag updates. It seems dovecot isn't always adding X-Status and X-UID headers (or updating the latter) when a message is read - I seem to have a mixture of "Status: O" or both headers missing. Is this because of mbox_dirty_syncs?
It didn't used to happen before today's fixes (but did happen before the very latest ones) Symptoms are Thunderbird (or Pine) think they've updated the flags, but when the folder is reopened, it reverts. I've tried deleting the dovecot.index* files.
Best Wishes, Chris
-- --+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+- Christopher Wakelin, c.d.wakelin@reading.ac.uk IT Services Centre, The University of Reading, Tel: +44 (0)118 378 8439 Whiteknights, Reading, RG6 2AF, UK Fax: +44 (0)118 975 3094
Pseudo mail caused this one too. I'll have to remember to start testing changes with mboxes having it too. Fixed it in 1.0-stable and tests.
But anyway if you have both UW-IMAP/Pine accessing the mbox at the same time as Dovecot, there can still be problems with flag changes getting lost. With mbox_lazy_writes=no you could get Dovecot to show changes to Pine earlier, but Pine doesn't have an option to do the same for Dovecot.
Those UID errors sound a bit troubling.. Hopefully they were fixed with this change. I don't think it should happen even if there's lots of mixed Dovecot/Pine/UW users accessing the same mbox..
On Tue, 2005-04-26 at 21:41 +0100, Chris Wakelin wrote:
I just looked in the logs (sorry, should have done that first) and it's been giving errors "mbox sync: UID inserted in the middle of mailbox .../INBOX (xxx > yyy)".
Chris
Chris Wakelin wrote:
Hmm, now I'm seeing problems with flag updates. It seems dovecot isn't always adding X-Status and X-UID headers (or updating the latter) when a message is read - I seem to have a mixture of "Status: O" or both headers missing. Is this because of mbox_dirty_syncs?
It didn't used to happen before today's fixes (but did happen before the very latest ones) Symptoms are Thunderbird (or Pine) think they've updated the flags, but when the folder is reopened, it reverts. I've tried deleting the dovecot.index* files.
Best Wishes, Chris
Timo Sirainen wrote:
Pseudo mail caused this one too. I'll have to remember to start testing changes with mboxes having it too. Fixed it in 1.0-stable and tests.
That seems better. Thanks for the quick response!
UW-IMAP still prefers to have pseudo-messages for efficiency, but doesn't create them if they're missing to prevent problems with some mail software that can't cope with it. I guess Dovecot prefers not to have them, as it's just as efficient without :)
But anyway if you have both UW-IMAP/Pine accessing the mbox at the same time as Dovecot, there can still be problems with flag changes getting lost. With mbox_lazy_writes=no you could get Dovecot to show changes to Pine earlier, but Pine doesn't have an option to do the same for Dovecot.
I've not been mixing UW and Dovecot on my INBOX, just on the AutoSpam folder. However, some of my colleagues who are helping me to test it have been switching between the two regularly; e.g. use Dovecot at work and UW-IMAP (via webmail) from home.
I'm hoping solving the UIDVALIDITY issue will make the switching back and forth from UW to Dovecot transparent for them (at the moment they have to open affected folders twice if UW has changed the UIDVALIDITY.)
I've warned them it may not be a good idea to have both servers accessing the same mailbox at the same time. (BTW is it OK to have two versions of Dovecot accessing it at the same time, e.g. 1.0-stable and 1.0-test68? I've assumed not!)
Those UID errors sound a bit troubling.. Hopefully they were fixed with this change. I don't think it should happen even if there's lots of mixed Dovecot/Pine/UW users accessing the same mbox..
They seemed to go away, so perhaps it was a transient thing.
Best Wishes, Chris
-- --+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+- Christopher Wakelin, c.d.wakelin@reading.ac.uk IT Services Centre, The University of Reading, Tel: +44 (0)118 378 8439 Whiteknights, Reading, RG6 2AF, UK Fax: +44 (0)118 975 3094
participants (3)
-
Chris Wakelin
-
Timo Sirainen
-
Todd Burroughs