[Dovecot] using dsync to convert mailboxes looses caching options

[Mark Zealey]

Apologies for top-posting but I can't figure out how to make this client do inline... I am seeing on the first run (we are using 'backup') we don't get any of the cache copied just the index files created. On the second run (ie when dest exists); a cache file is created and populated with the bits that are required for the sync presumably - guid. As you say the yes/tmp caching decisions are copied over (and visible in the cache file) but because the last used date is not copied; these fields are not activated for any of the messages so none of their data actually gets cached. I'm not seeing a compression at the end as the tmp etc fields are still there (mostly don't have any yes fields in our source caches) but as I say, because they don't have a last used date then the none of them are ever actually used until the client requests them via pop/imap.

Mark
________________________________________
From: Timo Sirainen [tss at iki.fi]
Sent: 08 December 2011 05:53
To: Mark Zealey
Cc: Dovecot Mailing List
Subject: Re: [Dovecot] using dsync to convert mailboxes looses caching options

On Sat, 2011-11-26 at 18:33 +0200, Mark Zealey wrote:

> We're trying to convert users from Maildir to sdbox at present; I'm
> using dsync to achieve this (2.0.16) however when the user's have been
> converted we only get minimal information in the caching files. Is there
> some way to preserve all the caching decisions that were previously made
> so that when the user logs in to the new mailbox we don't have to cause
> an io storm rebuilding the cache that we know was good? Dovecot seems to
> be partially doing this - if i remove the logs/cache from the source
> mailbox no cache files are built in the conversion; if i put them back
> then we get a cache file built but it only contains a few bits of
> information (guid, date.save). Looking into this a bit further i find
> that when the caches are present at source the fields are preserved but
> the 'last used' date and caching decisions are not which I suspect means
> dsync doesn't bother caching on import - only fields with a yes decision
> in the source are copied (but their decision is only copied as a tmp
> with the date of import). For example:

How are you calling dsync? Does the destination already exist? I tried
with:

rm -rf /tmp/foo; dsync -u tss -m INBOX mirror sdbox:/tmp/foo

It sets all of the cache fields with "yes" or "tmp" decision, as it
should. But yes, the "last used" field should probably be copied as
well.

Perhaps the problem with you is that dsync actually writes all of the
cache fields, but then it does a "cache compression" at the end, which
sees that the "last used" fields are so old, so it deletes them.

But yes, it is a problem that dsync doesn't update caching decisions..
Hmm. I guess I'll have to fix that for v2.1.