[Dovecot] Thunderbird message cache out of sync after repetitive rsyncs...
Ok, hopefully there is a solution to this.
I've been experimenting with multiple rsyncs in preparation for pulling the trigger on the mail server switch, but have a problem that I really want to fox before doing so.
Apparently something causes Thunderbirds local message cache to get out of sync with dovecot after a sync.
Here is the series of commands I'm running:
stop postfix, stop dovecot on new server
rsync -rltgovDHP --delete --exclude-from 'excludes.txt' /path/to/vmail/example.com/ /var/vmail/example.com/
chown vmail:vmail /var/vmail
start dovecot, start postfix
ls -al /var/vmail/example.com/user/cur
shows all of the messages that should be there, and all perms are correct.
Go to my account that is pointed to this mail server/account, and none of the new messages show up. Also, some messages are still showing up that shouldn't.
I've tried compacting the folders, closing/relaunching Thunderbird, but nothing helps.
The only way to get them to show up is to go to the local Thunderbird cache for the account, and delete the files associated with the folder having the problem.
The problem is, ALL folders will have this problem, which means that everyone will need to delete ALL of their local cahced folders.
This will be a major support problem.
Anyone have any ideas?
--
Best regards,
*/Charles/*
Am 24.12.2013 18:02, schrieb Charles Marcus:
Ok, hopefully there is a solution to this.
I've been experimenting with multiple rsyncs in preparation for pulling the trigger on the mail server switch, but have a problem that I really want to fox before doing so.
Apparently something causes Thunderbirds local message cache to get out of sync with dovecot after a sync
that is not dovecot specific and a thunderbird problem right click on the folder -> properties -> repair
it happens from time to time that after that messages re-appear and this happens on any mailserver, not only dovecot
On 2013-12-24 12:04 PM, Reindl Harald <h.reindl@thelounge.net> wrote:
Am 24.12.2013 18:02, schrieb Charles Marcus:
Apparently something causes Thunderbirds local message cache to get out of sync with dovecot after a sync
that is not dovecot specific and a thunderbird problem right click on the folder -> properties -> repair
it happens from time to time that after that messages re-appear and this happens on any mailserver, not only dovecot
Thanks, that won't be quite so bad - except for people who have dozens (some almost a hundred) folders...
I think it might be better to just delete the local cached copies of everything.
Do you know if there is an open bug for Thunderbird for this?
--
Best regards,
*/Charles/*
Am 24.12.2013 18:16, schrieb Charles Marcus:
On 2013-12-24 12:04 PM, Reindl Harald <h.reindl@thelounge.net> wrote:
Am 24.12.2013 18:02, schrieb Charles Marcus:
Apparently something causes Thunderbirds local message cache to get out of sync with dovecot after a sync
that is not dovecot specific and a thunderbird problem right click on the folder -> properties -> repair
it happens from time to time that after that messages re-appear and this happens on any mailserver, not only dovecot
Thanks, that won't be quite so bad - except for people who have dozens (some almost a hundred) folders... I think it might be better to just delete the local cached copies of everything.
- stop thunderbird
- delete any .msf file you find
- you are done
i am doing this once a year as well as for the "global-messages-db.sqlite" after i archive my current message structure below "2013" at the end of the year and let rebuild the whole caches
Do you know if there is an open bug for Thunderbird for this?
i doubt there is a way to debug this predictable
it happens AFAIK when different clients are changing the mailbox state at the same time and i doubt this only affects thunderbird, but only for thundebrir dteh global fix is possible that easy
On 12/24/2013 11:02 AM, Charles Marcus wrote:
Ok, hopefully there is a solution to this.
I've been experimenting with multiple rsyncs in preparation for pulling the trigger on the mail server switch, but have a problem that I really want to fox before doing so.
Apparently something causes Thunderbirds local message cache to get out of sync with dovecot after a sync.
Here is the series of commands I'm running:
stop postfix, stop dovecot on new server
rsync -rltgovDHP --delete --exclude-from 'excludes.txt' /path/to/vmail/example.com/ /var/vmail/example.com/
chown vmail:vmail /var/vmail
start dovecot, start postfix
ls -al /var/vmail/example.com/user/cur
shows all of the messages that should be there, and all perms are correct.
Go to my account that is pointed to this mail server/account, and none of the new messages show up. Also, some messages are still showing up that shouldn't.
I've tried compacting the folders, closing/relaunching Thunderbird, but nothing helps.
The only way to get them to show up is to go to the local Thunderbird cache for the account, and delete the files associated with the folder having the problem.
The problem is, ALL folders will have this problem, which means that everyone will need to delete ALL of their local cahced folders.
This will be a major support problem.
Anyone have any ideas?
The source of the problem is almost certainly out of sync Dovecot index files between the old and new servers, and thus TBird. After an rsync copy of the mails the new server must create the indexes on-the-fly when TBird connects, and the resulting new indexes are likely not identical to the old server. Thus TBird is seeing a different mailbox view.
TBird keeps its own indexes for all IMAP folders. It has nothing little or nothing to do with local cached copies of folders. I don't use GLODA and I don't cache locally, but I still have a .msf file for each Dovecot IMAP folder, some of them multiple MBs in size. These are strictly indexes. It's these local indexes not being in sync with your new Dovecot server indexes that I'm pretty sure is the cause of your problem.
If the mailbox contents are identical before/after the copy, you might try copying the indexes over from the old mail server, preserving permissions, creation time, atime, etc. If the server indexes are identical before/after the rsync you should avoid this problem, assuming everything else is identical, including server hostnames, IP addresses, encryption key, etc, etc. TBird tracks mailboxes by server name in Account Settings after all. If the server name changes that'll cause TBird to create an alternate local folder hierarchy in the profile directory. And that'll wreak havoc on your indexes, mailbox view, etc.
-- Stan
On 2013-12-24 1:11 PM, Stan Hoeppner <stan@hardwarefreak.com> wrote:
If the mailbox contents are identical before/after the copy, you might try copying the indexes over from the old mail server, preserving permissions, creation time, atime, etc.
Can't do this, because the old server doesn't have on disk indexes. It doesn't have enough disk space left (enabling indexing would probably cause the filesystem to fill up).
Disk space is one big reason for the migration - the other being that the old server is... well, so old (about 9 years now)...
:)
TBird tracks mailboxes by server name in Account Settings after all. If the server name changes that'll cause TBird to create an alternate local folder hierarchy in the profile directory. And that'll wreak havoc on your indexes, mailbox view, etc.
When I pull the trigger, I'll simple change the DNS pointers, so no, the server name will not change (from Thunderbird's perspective).
I'm mainly just interested in the end users not having to do anything to 'refresh' their local caches.
Maybe the best solution is a script that deletes all local copies of .msf files... although I had another thought I'll ask about in a renamed thread...
--
Best regards,
*/Charles/*
On 2013-12-24 12:02 PM, Charles Marcus <CMarcus@Media-Brokers.com> wrote:
Apparently something causes Thunderbirds local message cache to get out of sync with dovecot after a sync.
Ok, I had a new thought about the problem of invalid local client caches...
There is this on the dovecot wiki about the control files:
http://wiki2.dovecot.org/MailLocation/Maildir
"Dovecot stores some Maildir metadata into two control files:
dovecot-uidlist file contains IMAP UID <-> Maildir filename mapping... If the messages get new UIDs, the IMAP clients will invalidate their local cache and download the messages all over again. If you do this for all the users, you could cause huge disk I/O bursts to your server. "
This sounds like exactly what I want (IMAP clients will INVALIDATE THEIR LOCAL CACHE...) to happen the first time users access their mail on the new server.
So, I'm thinking what I could do is simply exclude the dovecot-uid* files during the transfer (and delete existing ones using --delete-excluded) during the rsync, and this would cause everyone's Thunderbird to regenerate their local caches, thus eliminating the need to rebuild the .msf files (either manually or by scripting their deletion)?
The new server is a VM on a pretty hefty box, with 16GB allocated, and if I do this before people come in, their initial logins would be spread out, not exactly at the same time, so the 'huge disk IO bursts' shouldn't be a big problem.
Obviously this would only happen the for the client they first access their mail with, so I'd need to block access to the new mail server from outside (ie, so mobile clients, which are polling the new server, wouldn't cause the regeneration of the dovecot-uid* files before Thunderbird gets a chance to, but...
Hmmm... what will happen for mobile clients... crap... will those get confused to, but leaving me with no easy way to delete their local cache (like I could the .msf files)?
Have to first test and make sure this will actually solve the problem though...
--
Best regards,
*/Charles/*
None of that should be happening. The client shouldn't be able to become confused, because it should see identical mailboxes in the old and in the new server. You can check with IMAP protocol if the rsync actually preserved everything correctly (what's in excludes.txt?) :
doveadm exec imap -u user@domain a select inbox b uid search all
Do this in both old and new server. Make sure that UIDVALIDITY and UIDNEXT replies to SELECT are the same, and also that the SEARCH reply is the same.
Did you stop Dovecot on the old server during rsync? That could explain if it was modified while rsync was running.
On 24.12.2013, at 19.02, Charles Marcus <CMarcus@Media-Brokers.com> wrote:
Ok, hopefully there is a solution to this.
I've been experimenting with multiple rsyncs in preparation for pulling the trigger on the mail server switch, but have a problem that I really want to fox before doing so.
Apparently something causes Thunderbirds local message cache to get out of sync with dovecot after a sync.
Here is the series of commands I'm running:
stop postfix, stop dovecot on new server
rsync -rltgovDHP --delete --exclude-from 'excludes.txt' /path/to/vmail/example.com/ /var/vmail/example.com/
chown vmail:vmail /var/vmail
start dovecot, start postfix
ls -al /var/vmail/example.com/user/cur
shows all of the messages that should be there, and all perms are correct.
Go to my account that is pointed to this mail server/account, and none of the new messages show up. Also, some messages are still showing up that shouldn't.
I've tried compacting the folders, closing/relaunching Thunderbird, but nothing helps.
The only way to get them to show up is to go to the local Thunderbird cache for the account, and delete the files associated with the folder having the problem.
The problem is, ALL folders will have this problem, which means that everyone will need to delete ALL of their local cahced folders.
This will be a major support problem.
Anyone have any ideas?
--
Best regards,
*/Charles/*
Hi Timo,
Thanks for the reply, but I think I figured it out...
I just realized...
the source does NOT have any index files at all, only the dovecot-uid* files
the target (the account I am testing with) *does* have *two* index files (but not the main dovecot.index file), because they are created when I access the account
So, maybe I just need to delete the index files on the target before accessing it?
I also really, really need to know why I don't have the main dovecot.index file... ???
I'll test the above hypothesis after the current backup finishes...
On 2013-12-26 11:05 AM, Timo Sirainen <tss@iki.fi> wrote:
None of that should be happening. The client shouldn't be able to become confused, because it should see identical mailboxes in the old and in the new server. You can check with IMAP protocol if the rsync actually preserved everything correctly (what's in excludes.txt?) :
doveadm exec imap -u user@domain a select inbox b uid search all
Do this in both old and new server. Make sure that UIDVALIDITY and UIDNEXT replies to SELECT are the same, and also that the SEARCH reply is the same.
Did you stop Dovecot on the old server during rsync? That could explain if it was modified while rsync was running.
On 24.12.2013, at 19.02, Charles Marcus <CMarcus@Media-Brokers.com> wrote:
Ok, hopefully there is a solution to this.
I've been experimenting with multiple rsyncs in preparation for pulling the trigger on the mail server switch, but have a problem that I really want to fox before doing so.
Apparently something causes Thunderbirds local message cache to get out of sync with dovecot after a sync.
Here is the series of commands I'm running:
stop postfix, stop dovecot on new server
rsync -rltgovDHP --delete --exclude-from 'excludes.txt' /path/to/vmail/example.com/ /var/vmail/example.com/
chown vmail:vmail /var/vmail
start dovecot, start postfix
ls -al /var/vmail/example.com/user/cur
shows all of the messages that should be there, and all perms are correct.
Go to my account that is pointed to this mail server/account, and none of the new messages show up. Also, some messages are still showing up that shouldn't.
I've tried compacting the folders, closing/relaunching Thunderbird, but nothing helps.
The only way to get them to show up is to go to the local Thunderbird cache for the account, and delete the files associated with the folder having the problem.
The problem is, ALL folders will have this problem, which means that everyone will need to delete ALL of their local cahced folders.
This will be a major support problem.
Anyone have any ideas?
--
Best regards,
*/Charles /*
On 2013-12-26 1:33 PM, Charles Marcus <CMarcus@Media-Brokers.com> wrote:
the source does NOT have any index files at all, only the dovecot-uid* files
the target (the account I am testing with) *does* have *two* index files (but not the main dovecot.index file), because they are created when I access the account
So, maybe I just need to delete the index files on the target before accessing it?
Meaning... the index files and the dovecot-uid* files are out of sync (index files are in a state from the last time I accessed the mailstore, before I rsync'd it again, but the dovecot-uid* files are from the state of the mailstore at the time of the last backup).
Would that explain it?
--
Best regards,
*/Charles/*
On 26.12.2013, at 20.40, Charles Marcus <CMarcus@Media-Brokers.com> wrote:
On 2013-12-26 1:33 PM, Charles Marcus <CMarcus@Media-Brokers.com> wrote:
the source does NOT have any index files at all, only the dovecot-uid* files
the target (the account I am testing with) *does* have *two* index files (but not the main dovecot.index file), because they are created when I access the account
So, maybe I just need to delete the index files on the target before accessing it?
Meaning... the index files and the dovecot-uid* files are out of sync (index files are in a state from the last time I accessed the mailstore, before I rsync'd it again, but the dovecot-uid* files are from the state of the mailstore at the time of the last backup).
Would that explain it?
Ah. Maybe. Although they do have timestamps and should refresh automatically if they don't match. But I've heard some problems related to that. Using doveadm force-resync -A '*' could be used to make sure the indexes are up to date for everyone.
Am 26.12.2013 19:45, schrieb Timo Sirainen:
On 26.12.2013, at 20.40, Charles Marcus <CMarcus@Media-Brokers.com> wrote:
On 2013-12-26 1:33 PM, Charles Marcus <CMarcus@Media-Brokers.com> wrote:
the source does NOT have any index files at all, only the dovecot-uid* files
the target (the account I am testing with) *does* have *two* index files (but not the main dovecot.index file), because they are created when I access the account
So, maybe I just need to delete the index files on the target before accessing it?
Meaning... the index files and the dovecot-uid* files are out of sync (index files are in a state from the last time I accessed the mailstore, before I rsync'd it again, but the dovecot-uid* files are from the state of the mailstore at the time of the last backup).
Would that explain it?
Ah. Maybe. Although they do have timestamps and should refresh automatically if they don't match. But I've heard some problems related to that. Using doveadm force-resync -A '*' could be used to make sure the indexes are up to date for everyone.
off topic, i sometimes have bugs with Thunderbird on Windows only ( no idea why ) only dovecot-uid* delete fixes that, and all mails in dove maildir can be seen again on TB win, never had that problem with thunderbird on linux.
Best Regards MfG Robert Schetterer
-- [*] sys4 AG
http://sys4.de, +49 (89) 30 90 46 64 Franziskanerstraße 15, 81669 München
Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263 Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer Aufsichtsratsvorsitzender: Florian Kirstein
On 2013-12-26 2:02 PM, Robert Schetterer <rs@sys4.de> wrote:
Am 26.12.2013 19:45, schrieb Timo Sirainen:
On 26.12.2013, at 20.40, Charles Marcus <CMarcus@Media-Brokers.com> wrote:
Meaning... the index files and the dovecot-uid* files are out of sync (index files are in a state from the last time I accessed the mailstore, before I rsync'd it again, but the dovecot-uid* files are from the state of the mailstore at the time of the last backup).
Would that explain it?
Ah. Maybe. Although they do have timestamps and should refresh automatically if they don't match. But I've heard some problems related to that. Using doveadm force-resync -A '*' could be used to make sure the indexes are up to date for everyone.
off topic, i sometimes have bugs with Thunderbird on Windows only ( no idea why ) only dovecot-uid* delete fixes that, and all mails in dove maildir can be seen again on TB win, never had that problem with thunderbird on linux.
Ok, I'll test two different ways...
First, I'll test if just deleting the index files on the target before accessing with Thunderbird shows all files.
If it doesn't, then I'll test if deleting both the indexes and the uid* files does the trick.
Thanks,
--
Best regards,
*/Charles/*
Am 26.12.2013 21:21, schrieb Charles Marcus:
On 2013-12-26 2:02 PM, Robert Schetterer <rs@sys4.de> wrote:
Am 26.12.2013 19:45, schrieb Timo Sirainen:
On 26.12.2013, at 20.40, Charles Marcus <CMarcus@Media-Brokers.com> wrote:
Meaning... the index files and the dovecot-uid* files are out of sync (index files are in a state from the last time I accessed the mailstore, before I rsync'd it again, but the dovecot-uid* files are from the state of the mailstore at the time of the last backup).
Would that explain it?
Ah. Maybe. Although they do have timestamps and should refresh automatically if they don't match. But I've heard some problems related to that. Using doveadm force-resync -A '*' could be used to make sure the indexes are up to date for everyone.
off topic, i sometimes have bugs with Thunderbird on Windows only ( no idea why ) only dovecot-uid* delete fixes that, and all mails in dove maildir can be seen again on TB win, never had that problem with thunderbird on linux.
Ok, I'll test two different ways...
First, I'll test if just deleting the index files on the target before accessing with Thunderbird shows all files.
If it doesn't, then I'll test if deleting both the indexes and the uid* files does the trick.
in my case this has worked ever in the "magic TB win list mail bug", but it might be an additional problem not directly related to your problem
Thanks,
Best Regards MfG Robert Schetterer
-- [*] sys4 AG
http://sys4.de, +49 (89) 30 90 46 64 Franziskanerstraße 15, 81669 München
Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263 Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer Aufsichtsratsvorsitzender: Florian Kirstein
Just closing this related thread too...
Again apologies to all.
This is my first live server migration, so being very careful not to make any mistakes (especially involving losing mail)...
On 2013-12-26 3:21 PM, Charles Marcus <CMarcus@Media-Brokers.com> wrote:
Ok, I'll test two different ways...
First, I'll test if just deleting the index files on the target before accessing with Thunderbird shows all files.
If it doesn't, then I'll test if deleting both the indexes and the uid* files does the trick.
--
Best regards,
*/Charles/*
On 26.12.2013, at 20.33, Charles Marcus <CMarcus@Media-Brokers.com> wrote:
Hi Timo,
Thanks for the reply, but I think I figured it out...
I just realized...
- the source does NOT have any index files at all, only the dovecot-uid* files
Only the dovecot-uidlist matters, the indexes are only for optimization.
- the target (the account I am testing with) *does* have *two* index files (but not the main dovecot.index file), because they are created when I access the account
This is intentional optimization.
So, maybe I just need to delete the index files on the target before accessing it?
Nope, doesn't matter.
I also really, really need to know why I don't have the main dovecot.index file... ???
It's going to be created later once the mailbox has had a bit more changes.
On 2013-12-26 1:44 PM, Timo Sirainen <tss@iki.fi> wrote:
On 26.12.2013, at 20.33, Charles Marcus <CMarcus@Media-Brokers.com> wrote:
I also really, really need to know why I don't have the main dovecot.index file... ???
It's going to be created later once the mailbox has had a bit more changes.
Oh... ok... didn't see anything on the wiki about a delay with this file being created.
Thanks!
participants (5)
-
Charles Marcus
-
Reindl Harald
-
Robert Schetterer
-
Stan Hoeppner
-
Timo Sirainen