Re: [Dovecot] using dsync to convert mailboxes looses caching options
With 2.0.16 hdr.xxx fields get copied fine (but of course without timestamp). With the patch you provided they don't get copied whether using mirror or backup & starting from scratch. I'm doing a Maildir to sdbox migration otherwise don't think I'm doing anytihng strange.
Mark
From: Mark Zealey Sent: 08 December 2011 09:35 To: Timo Sirainen Cc: Dovecot Mailing List Subject: RE: [Dovecot] using dsync to convert mailboxes looses caching options
OK I'll test the header copying more fully. The reason we want to preserve caching decisions is to avoid an IO storm when users log in to their mailboxes after an sdbox upgrade so it would be great to be able to have some way to warm caches.
Mark
On Thu, 2011-12-08 at 14:45 +0000, Mark Zealey wrote:
With 2.0.16 hdr.xxx fields get copied fine (but of course without timestamp). With the patch you provided they don't get copied whether using mirror or backup & starting from scratch. I'm doing a Maildir to sdbox migration otherwise don't think I'm doing anytihng strange.
Show the whole list of cache decisions in source and destination?
10-12-2011 08:28, Timo Sirainen yazmış:
On Thu, 2011-12-08 at 14:45 +0000, Mark Zealey wrote:
With 2.0.16 hdr.xxx fields get copied fine (but of course without timestamp). With the patch you provided they don't get copied whether using mirror or backup& starting from scratch. I'm doing a Maildir to sdbox migration otherwise don't think I'm doing anytihng strange. Show the whole list of cache decisions in source and destination?
Ah-ha it's doing the same in 2.0.16 - looking deeper it's because i havn't accessed the tmp fields in a week or two so I guess the decision has been taken not to migrate them.
Mark
On 10.12.2011, at 13.03, Mark Zealey wrote:
Ah-ha it's doing the same in 2.0.16 - looking deeper it's because i havn't accessed the tmp fields in a week or two so I guess the decision has been taken not to migrate them.
Yes, most likely the reason. Could this also explain the date.saved?
It could well be because of the conversion to sdbox then - the ctime/mtime of the files are not being preserved by dsync (in stock 2.0.16). The date.saved timestamp is only put into the cache on the second dsync run; presumably therefore it picks it up from the filesystem.
With sdbox the file's mtime isn't even tried to be preserved. The received-time and saved-time are written to the metadata block inside the file.
10-12-2011 13:07, Timo Sirainen yazmış:
It could well be because of the conversion to sdbox then - the ctime/mtime of the files are not being preserved by dsync (in stock 2.0.16). The date.saved timestamp is only put into the cache on the second dsync run; presumably therefore it picks it up from the filesystem. With sdbox the file's mtime isn't even tried to be preserved. The received-time and saved-time are written to the metadata block inside the file.
Ah yes; I saw the R metadata but not the C header key. Looking deeper at this I think I was expecting the date.save time to be about the same as the date.receive; however the ctime for these files is quite recent presumably affected by setting of message flags in a maildir or something (we're using nfs). The source cache says:
- date.received: 1301978447 (4f9d9a4d)
- date.save: 1322465550 (0e39d34e)
The message file itself has mtime 1301978447 and ctime 1323514077; and in the sdbox header/metadata we have:
C4ee3391a R4d9a9d4f
so ctime/sdbox C entry are close enough by my calculations (not sure where the 61 seconds of difference comes from though). It is a bit strange you wouldn't use the source cache's value for date.save if it is available as ctime can be pretty unreliable?
Mark
On 10.12.2011, at 13.32, Mark Zealey wrote:
10-12-2011 13:07, Timo Sirainen yazmış:
It could well be because of the conversion to sdbox then - the ctime/mtime of the files are not being preserved by dsync (in stock 2.0.16). The date.saved timestamp is only put into the cache on the second dsync run; presumably therefore it picks it up from the filesystem. With sdbox the file's mtime isn't even tried to be preserved. The received-time and saved-time are written to the metadata block inside the file.
Ah yes; I saw the R metadata but not the C header key.
The C is the file's create time. It's not actually use for anything.
Looking deeper at this I think I was expecting the date.save time to be about the same as the date.receive; however the ctime for these files is quite recent presumably affected by setting of message flags in a maildir or something (we're using nfs).
Yes, maildir flag changes change the ctime, which also changes the save date if it's not already cached.
so ctime/sdbox C entry are close enough by my calculations (not sure where the 61 seconds of difference comes from though). It is a bit strange you wouldn't use the source cache's value for date.save if it is available as ctime can be pretty unreliable?
It is using the the cached value.
Anyway, I remembered wrong how sdbox's save date is looked up. It's taken from the sdbox file's ctime. The reason is similar to maildir: The save date is used mainly to figure out when to automatically expunge messages from Trash after it's been there for n days. So if you copy 1 year old message to Trash, you don't want it expunged immediately (based on mtime or some metadata inside the file), you want it expunged n days since the move. And ctime is really the only nice way to do it automatically, because copying a message with sdbox is done with hard linking.
mdbox stores the save date in the index file. sdbox could do it too, but that's just extra work and probably not worth the trouble. And unlink atime/mtime, ctime can't be changed using any syscalls (except to current time).
So, I think everything here works as intended, although not really as expected. :)
participants (3)
-
Mark Zealey
-
Mark Zealey
-
Timo Sirainen