[Dovecot] Test migration (IMAP copy) and INTERNALDATE?
We're trying some migration tests ... from CommuniGate to Dovecot. The migration program does an APPEND into the new mailbox:
src>: a0 FETCH 2900 (FLAGS INTERNALDATE RFC822.PEEK) src<: * 2900 FETCH (FLAGS (\Seen) INTERNALDATE "29-Aug-2005 18:36:54 +0000" RFC822 {1194157} src<: ) dst>: bAPP APPEND "INBOX" (\Seen) "29-Aug-2005 18:36:54 +0000" {1194157} dst<: + OK
It appears to be trying to set the message date... in this example, to 29 Aug 2005.
Dovecot seems to be ignoring it.
Am I doing something wrong, or does Dovecot not allow this?
Steve
On Tue, 2006-04-18 at 10:02 -0400, Apps Lists wrote:
We're trying some migration tests ... from CommuniGate to Dovecot. The migration program does an APPEND into the new mailbox:
src>: a0 FETCH 2900 (FLAGS INTERNALDATE RFC822.PEEK) src<: * 2900 FETCH (FLAGS (\Seen) INTERNALDATE "29-Aug-2005 18:36:54 +0000" RFC822 {1194157} src<: ) dst>: bAPP APPEND "INBOX" (\Seen) "29-Aug-2005 18:36:54 +0000" {1194157} dst<: + OK
It appears to be trying to set the message date... in this example, to 29 Aug 2005.
Dovecot seems to be ignoring it.
Am I doing something wrong, or does Dovecot not allow this?
With maildir? Seems to be working with me..:
x append "INBOX" (\Seen) "29-Aug-2005 18:36:54 +0000" {10}
- OK From: foo
- 40 EXISTS
- 1 RECENT x OK Append completed. x fetch * internaldate
- 40 FETCH (INTERNALDATE "29-Aug-2005 21:36:54 +0300") x OK Fetch completed.
The date is stored in file's mtime field. What filesystem do you use? Maybe it's breaking this somehow.
On Tue, 2006-04-18 at 10:02 -0400, Apps Lists wrote:
We're trying some migration tests ... from CommuniGate to Dovecot. The migration program does an APPEND into the new mailbox:
src>: a0 FETCH 2900 (FLAGS INTERNALDATE RFC822.PEEK) src<: * 2900 FETCH (FLAGS (\Seen) INTERNALDATE "29-Aug-2005 18:36:54 +0000" RFC822 {1194157} src<: ) dst>: bAPP APPEND "INBOX" (\Seen) "29-Aug-2005 18:36:54 +0000" {1194157} dst<: + OK
It appears to be trying to set the message date... in this example, to 29 Aug 2005.
Dovecot seems to be ignoring it.
Am I doing something wrong, or does Dovecot not allow this?
With maildir? Seems to be working with me..:
x append "INBOX" (\Seen) "29-Aug-2005 18:36:54 +0000" {10}
- OK From: foo
- 40 EXISTS
- 1 RECENT x OK Append completed. x fetch * internaldate
- 40 FETCH (INTERNALDATE "29-Aug-2005 21:36:54 +0300") x OK Fetch completed.
The date is stored in file's mtime field. What filesystem do you use? Maybe it's breaking this somehow.
Hi Timo.
Wow, that's a strange one.
The test system is Centos 4.3 (Final) running kernel 2.6.9-34.ELsmp #1 SMP.
I tested this with NFS and local ext3. The mtime is initially set properly, but gets changed to current system time sometime later on in the copy process. I moved the mtime change code around and managed to work around this, although I'm sure there's a more elegant approach.
The patch is here on the list.
Steve
On Mon, 2006-04-24 at 07:47 -0400, Apps Lists wrote:
The test system is Centos 4.3 (Final) running kernel 2.6.9-34.ELsmp #1 SMP.
I tested this with NFS and local ext3. The mtime is initially set properly, but gets changed to current system time sometime later on in the copy process. I moved the mtime change code around and managed to work around this, although I'm sure there's a more elegant approach.
The patch is here on the list.
Umm. Where?
On Mon, 2006-04-24 at 07:47 -0400, Apps Lists wrote:
The test system is Centos 4.3 (Final) running kernel 2.6.9-34.ELsmp #1 SMP.
I tested this with NFS and local ext3. The mtime is initially set properly, but gets changed to current system time sometime later on in the copy process. I moved the mtime change code around and managed to work around this, although I'm sure there's a more elegant approach.
The patch is here on the list.
Umm. Where?
It did indeed make it to the list... Let me know if you don't have it and I'll resend privately.
Subject: [patch] Re: [Dovecot] Test migration (IMAP copy) and INTERNALDATE?] From: "Apps Lists" appslists@neonova.net Add Date: Wed, April 19, 2006 1:00 pm To: dovecot@dovecot.org
Steve
On Mon, 2006-04-24 at 10:40 -0400, Apps Lists wrote:
On Mon, 2006-04-24 at 07:47 -0400, Apps Lists wrote:
The test system is Centos 4.3 (Final) running kernel 2.6.9-34.ELsmp #1 SMP.
I tested this with NFS and local ext3. The mtime is initially set properly, but gets changed to current system time sometime later on in the copy process. I moved the mtime change code around and managed to work around this, although I'm sure there's a more elegant approach.
The patch is here on the list.
Umm. Where?
It did indeed make it to the list... Let me know if you don't have it and I'll resend privately.
Ok, saw it now. But doing it after link() is a bit too late. It's possible that another process could have seen the file at that point with the wrong timestamp. Also I don't think link() should update mtime, and it doesn't with me, neither with ext3 nor with NFS.
What if you try it manually?
stat somefile ln somefile somefile2 stat somefile2
Only "Change" time should have changed, "Access" and "Modify" should be the same. Or if you don't have stat, I guess ls -l is enough as long as the mtime is old enough..
On Mon, 2006-04-24 at 10:40 -0400, Apps Lists wrote:
The test system is Centos 4.3 (Final) running kernel 2.6.9-34.ELsmp #1 SMP.
I tested this with NFS and local ext3. The mtime is initially set
On Mon, 2006-04-24 at 07:47 -0400, Apps Lists wrote: properly, but gets changed to current system time sometime later on in the copy process. I moved the mtime change code around and managed to work around this, although I'm sure there's a more elegant approach.
The patch is here on the list.
Umm. Where?
It did indeed make it to the list... Let me know if you don't have it and I'll resend privately.
Ok, saw it now. But doing it after link() is a bit too late. It's possible that another process could have seen the file at that point with the wrong timestamp. Also I don't think link() should update mtime, and it doesn't with me, neither with ext3 nor with NFS.
What if you try it manually?
stat somefile ln somefile somefile2 stat somefile2
Only "Change" time should have changed, "Access" and "Modify" should be the same. Or if you don't have stat, I guess ls -l is enough as long as the mtime is old enough..
Hi Timo.
I knew I was introducing a race condition, but needed a quick fix for testing. This isn't producton yet. :)
I also did the same tests you recommended prior to making the source change. The ln doesn't seem to affect mtime. There has to be something else going on... what, I have no idea.
For giggles, the "ln test" is below.
Steve
root@dev3.neonova.net:/root# touch dovecot-test
root@dev3.neonova.net:/root# stat dovecot-test
File: dovecot-test' Size: 0 Blocks: 0 IO Block: 4096 regular empty file Device: fd00h/64768d Inode: 16402 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2006-04-25 07:16:41.000000000 -0400 Modify: 2006-04-25 07:16:41.000000000 -0400 Change: 2006-04-25 07:16:41.000000000 -0400 root@dev3.neonova.net:/root# ln dovecot-test test-dovecot root@dev3.neonova.net:/root# stat test-dovecot File: test-dovecot'
Size: 0 Blocks: 0 IO Block: 4096 regular empty
file
Device: fd00h/64768d Inode: 16402 Links: 2
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2006-04-25 07:16:41.000000000 -0400
Modify: 2006-04-25 07:16:41.000000000 -0400
Change: 2006-04-25 07:16:58.000000000 -0400
root@dev3.neonova.net:/root#
On Mon, 2006-04-24 at 10:40 -0400, Apps Lists wrote:
The test system is Centos 4.3 (Final) running kernel 2.6.9-34.ELsmp #1 SMP.
I tested this with NFS and local ext3. The mtime is initially set
On Mon, 2006-04-24 at 07:47 -0400, Apps Lists wrote: properly, but gets changed to current system time sometime later on in the copy process. I moved the mtime change code around and managed to work around this, although I'm sure there's a more elegant approach.
The patch is here on the list.
Umm. Where?
It did indeed make it to the list... Let me know if you don't have it and I'll resend privately.
Ok, saw it now. But doing it after link() is a bit too late. It's possible that another process could have seen the file at that point with the wrong timestamp. Also I don't think link() should update mtime, and it doesn't with me, neither with ext3 nor with NFS.
What if you try it manually?
stat somefile ln somefile somefile2 stat somefile2
Only "Change" time should have changed, "Access" and "Modify" should be the same. Or if you don't have stat, I guess ls -l is enough as long as the mtime is old enough..
Hi Timo.
I knew I was introducing a race condition, but needed a quick fix for testing. This isn't producton yet. :)
I also did the same tests you recommended prior to making the source change. The ln doesn't seem to affect mtime. There has to be something else going on... what, I have no idea.
For giggles, the "ln test" is below.
Steve
root@dev3.neonova.net:/root# touch dovecot-test root@dev3.neonova.net:/root# stat dovecot-test File:
dovecot-test' Size: 0 Blocks: 0 IO Block: 4096 regular empty file Device: fd00h/64768d Inode: 16402 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2006-04-25 07:16:41.000000000 -0400 Modify: 2006-04-25 07:16:41.000000000 -0400 Change: 2006-04-25 07:16:41.000000000 -0400 root@dev3.neonova.net:/root# ln dovecot-test test-dovecot root@dev3.neonova.net:/root# stat test-dovecot File:test-dovecot' Size: 0 Blocks: 0 IO Block: 4096 regular empty file Device: fd00h/64768d Inode: 16402 Links: 2 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2006-04-25 07:16:41.000000000 -0400 Modify: 2006-04-25 07:16:41.000000000 -0400 Change: 2006-04-25 07:16:58.000000000 -0400 root@dev3.neonova.net:/root#
Timo,
In an earlier message:
[bug?] Re: [Dovecot] Test migration (IMAP copy) and INTERNALDATE?
.... I had introduced some debug output with i_error() and put in a few calls to sleep() in order to determine if the utime() calls were actually working.
For example, I'd dump the filename in debug, then sleep 20 seconds prior to the call to utime(), after the call and after the link.
Watching the log and running stat/ls on the file during the sleep calls indicated that utime() indeed worked, but something changed the mtime time to current time after the mtime was set with utime().
I'm not entirely comfortable or familiar with the sources yet, so I just figured there was some other system call that was squashing the mtime. I wasn't able to find it.
Steve
participants (2)
-
Apps Lists
-
Timo Sirainen