[Dovecot] dsync mbox->mdbox II: highest_modseq changed
Hi again,
after replacing CRLF in 4 mboxes dsync was able to sync all of the 30+GB mailstore to mdbox.
Now the (repeated) calling of dsync yields info messages of the kind
dsync(athimm): Info: old/speicher: highest_modseq changed: 1 != 10 dsync(athimm): Info: root/root-heretic: highest_modseq changed: 1 != 10 dsync(athimm): Info: lists/ccrma.stanford.edu/planetccrma: highest_modseq changed: 11 != 14
I actually have 987 such messages for a total of 1115 mboxes. I checked a couple of them to see whether this is a CRLF issue and found no CRLF, so it's probably something different.
As long as the mboxes were still being delivered to or read from I thought it was normal, but I'd like to see a silent dsync which would give me more confidence in the migration process. So I kept from delivering to the mbox store as well as reading from it and performed multiple dsyncs.
The output of dsync became stable, e.g. the "highest_modseq changed" messages were exactly the same. Before nuking all indexes from the mbox storage and the mdbox itself and then repeating the whole process (which does take a long time), I wanted to know whether I should debug something.
Thanks!
On Thu, 2010-11-18 at 18:57 +0200, Axel Thimm wrote:
dsync(athimm): Info: old/speicher: highest_modseq changed: 1 != 10 dsync(athimm): Info: root/root-heretic: highest_modseq changed: 1 != 10 dsync(athimm): Info: lists/ccrma.stanford.edu/planetccrma: highest_modseq changed: 11 != 14
These don't really matter. It just couldn't sync the modseqs correctly. I should try to fix those somehow some day, but I think the problem here is that mbox code can't handle this correctly. It should have increased the modseqs to 10, 10, 14 but apparently it didn't.
Hi,
On Thu, 2010-11-18 at 17:34 +0000, Timo Sirainen wrote:
On Thu, 2010-11-18 at 18:57 +0200, Axel Thimm wrote:
dsync(athimm): Info: old/speicher: highest_modseq changed: 1 != 10 dsync(athimm): Info: root/root-heretic: highest_modseq changed: 1 != 10 dsync(athimm): Info: lists/ccrma.stanford.edu/planetccrma: highest_modseq changed: 11 != 14
These don't really matter. It just couldn't sync the modseqs correctly. I should try to fix those somehow some day, but I think the problem here is that mbox code can't handle this correctly. It should have increased the modseqs to 10, 10, 14 but apparently it didn't.
Shouldn't it be the other way? mbox was the source, so its modseq should had been considered authoritative, or not? One would expect that after applying the (same) changes to mdbox that the modseq should converge to be the same.
I'm just trying to understand whether the bug may be in the mdbox part of the code. On another note I now switched to mdbox with bzip compressed storage and everything seems nice and smooth. The server has a much lower load when I connect and even the client I'm using (evolution) which would previously crash due to timeouts is now working much better (though still crashing, but that's not really related to dovecot). The storage requirements have more than halved (15GB vs 34GB) and are much easier to rsync/backup. I love mdbox. :)
On 2010-11-19 1:40 AM, Axel Thimm wrote:
The storage requirements have more than halved (15GB vs 34GB) and are much easier to rsync/backup. I love mdbox. :)
Hi Axel,
Are you using SiS? Or was this just due to having far fewer tiny files wasting blocks (or however that works)?
--
Best regards,
Charles
On 19.11.2010, at 6.40, Axel Thimm wrote:
dsync(athimm): Info: old/speicher: highest_modseq changed: 1 != 10 dsync(athimm): Info: root/root-heretic: highest_modseq changed: 1 != 10 dsync(athimm): Info: lists/ccrma.stanford.edu/planetccrma: highest_modseq changed: 11 != 14
These don't really matter. It just couldn't sync the modseqs correctly. I should try to fix those somehow some day, but I think the problem here is that mbox code can't handle this correctly. It should have increased the modseqs to 10, 10, 14 but apparently it didn't.
Shouldn't it be the other way? mbox was the source, so its modseq should had been considered authoritative, or not? One would expect that after applying the (same) changes to mdbox that the modseq should converge to be the same.
The problem is that modseqs can't shrink. Also once modseqs are enabled, whenever something changes the modseqs always increase. With mbox you didn't have modseqs enabled, so highest_modseq=1 at the beginning of dsync. But dsync enables modseqs and so it can't preserve the modseq=1 for destination, because each change increases it.. It would of course be possible to delay enabling modseqs or maybe not enable them at all, but that's a lot more extra complexity..
participants (3)
-
Axel Thimm
-
Charles Marcus
-
Timo Sirainen