[Dovecot] Weird internaldate behavior with 1.0-test80 and mbox folder
Has anyone else encountered weird date behavior with mbox-style folders?
I am nearly ready to bite the bullet and convert my UW imap hierarchy (which is in mbx format) to Dovecot. In so doing, I need to convert each UW-specific mbx-format mailboxes to the more portable mbox format, which Dovecot can read. (This is done with the 'mbxcvt' utility that's part of UW.)
I am then planning to copy those mbox-format folders into a staging area in a directory I've defined to Dovecot as a mbox-format namespace. The plan was then to use my normal GUI to reorganize my old data and copy it into proper maildir folders into Dovecot's default namespace.
Since I'm naturally paranoid, I did a dry run of a couple folders first and discovered a rather strange anomaly -- all the mail in the converted folder is displayed by Dovecot with a timestamp exactly 200 minutes (3 hours 20 minutes) into the future!
Investigating this via telneting to the respective IMAP ports shows that UW imapd shows the correct timestamp both in the original mbx-format folder and the converted mbox-format folder -- so it appears that Dovecot is skewing the time.
This doesn't look like a timezone issue since 200 minutes ahead from CST6CDT would be a pretty strange timezone -- not even Newfoundland has this odd a TZ value! :)
(For the curious, Newfoundland is a Canadian province whose
time zone is TZ=NST3:30NDT, therefore being a half-hour "off"
from the rest of North America.)I have distilled this into a degenerate case of a single folder with a single message and can reproduce. As mentioned, I am running 1.0-test80. My platform is AIX 5.1.
I created the test by sending a message to myself using
date | mail -s "Test at $(date)" roma@uiuc.eduand using my MUA (Mulberry) to file it into the test folder, which defaults to mbx format in UW's namespace. Here's what UW shows when I select/fetch the "original" message:
- namespace
- NAMESPACE (("" "/")("#mhinbox" NIL)("#mh/" "/")) (("~" "/")) (("#shared/" "/")("#ftp/" "/")("#news." ".")("#public/" "/"))
- OK NAMESPACE completed
- select "~/Test"
- 1 EXISTS
- 0 RECENT
- OK [UIDVALIDITY 1124811965] UID validity status
- OK [UIDNEXT 3] Predicted next UID
- FLAGS (\Answered \Flagged \Deleted \Draft \Seen)
- OK [PERMANENTFLAGS (\* \Answered \Flagged \Deleted \Draft \Seen)] Permanent flags
- OK [READ-WRITE] SELECT completed
- fetch 1 internaldate
- 1 FETCH (INTERNALDATE "23-Aug-2005 11:16:37 -0500")
- OK FETCH completed
- fetch 1 envelope
- 1 FETCH (ENVELOPE ("Tue, 23 Aug 2005 11:16:30 -0500" "Test at Tue Aug 23 11:16:29 CDT 2005" (("Jon Roma" NIL "roma" "zippy.cso.uiuc.edu")) (("Jon Roma" NIL "roma" "zippy.cso.uiuc.edu")) (("Jon Roma" NIL "roma" "zippy.cso.uiuc.edu")) ((NIL NIL "roma" "uiuc.edu")) NIL NIL NIL "200508231616.j7NGGUrc022618@zippy.cso.uiuc.edu"))
- OK FETCH completed
Next I use the UW 'mbxcvt' to convert the folder, leaving it in mbox format in a different folder (~/Test-mbox) -- still in UW's namespace. Here's what the telnet connection shows:
- namespace
- NAMESPACE (("" "/")("#mhinbox" NIL)("#mh/" "/")) (("~" "/")) (("#shared/" "/")("#ftp/" "/")("#news." ".")("#public/" "/"))
- OK NAMESPACE completed
- select "~/Test-mbox"
- 1 EXISTS
- 1 RECENT
- OK [UIDVALIDITY 1124813840] UID validity status
- OK [UIDNEXT 2] Predicted next UID
- FLAGS (\Answered \Flagged \Deleted \Draft \Seen)
- OK [PERMANENTFLAGS (\* \Answered \Flagged \Deleted \Draft \Seen)] Permanent flags
- OK [READ-WRITE] SELECT completed
- fetch 1 internaldate
- 1 FETCH (INTERNALDATE "23-Aug-2005 11:16:37 -0500")
- OK FETCH completed
- fetch 1 envelope
- 1 FETCH (ENVELOPE ("Tue, 23 Aug 2005 11:16:30 -0500" "Test at Tue Aug 23 11:16:29 CDT 2005" (("Jon Roma" NIL "roma" "zippy.cso.uiuc.edu")) (("Jon Roma" NIL "roma" "zippy.cso.uiuc.edu")) (("Jon Roma" NIL "roma" "zippy.cso.uiuc.edu")) ((NIL NIL "roma" "uiuc.edu")) NIL NIL NIL "200508231616.j7NGGUrc022618@zippy.cso.uiuc.edu"))
- OK FETCH completed
Now, copying the same mbox-format folder into dovecot's namespace and telneting to my Dovecot port gives the following:
- namespace
- NAMESPACE (("" ".")("Stage/" "/")) NIL NIL
- OK Namespace completed.
- select "Stage/Test"
- FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
- OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft \*)] Flags permitted.
- 1 EXISTS
- 1 RECENT
- OK [UIDVALIDITY 1124815188] UIDs valid
- OK [UIDNEXT 2] Predicted next UID
- OK [READ-WRITE] Select completed.
- fetch 1 internaldate
- 1 FETCH (INTERNALDATE "23-Aug-2005 14:36:37 -0500")
- OK Fetch completed.
- fetch 1 envelope
- 1 FETCH (ENVELOPE ("Tue, 23 Aug 2005 11:16:30 -0500" "Test at Tue Aug 23 11:16:29 CDT 2005" (("Jon Roma" NIL "roma" "zippy.cso.uiuc.edu")) (("Jon Roma" NIL "roma" "zippy.cso.uiuc.edu")) (("Jon Roma" NIL "roma" "zippy.cso.uiuc.edu")) ((NIL NIL "roma" "uiuc.edu")) NIL NIL NIL "200508231616.j7NGGUrc022618@zippy.cso.uiuc.edu"))
- OK Fetch completed.
Notice the "internaldate" response. Dovecot reports the correct time zone, but the internaldate value is 200 minutes ahead of what it should be!!!
This is an interesting mystery; I seem to be living three hours and 20 minutes in the future.
I've attached the entire contents of the mbox-style mailbox that Dovecot misinterprets.
Thanks in advance.
From MAILER-DAEMON Tue Aug 23 11:39:48 2005Date: 23 Aug 2005 11:39:48 -0500 From: Mail System Internal Data MAILER-DAEMON@zippy.cso.uiuc.edu Subject: DON'T DELETE THIS MESSAGE -- FOLDER INTERNAL DATA X-IMAP: 1124815188 0000000001 Status: RO
This text is part of the internal format of your mail folder, and is not a real message. It is created automatically by the mail system software. If deleted, important folder data will be lost, and it will be re-created with the data reset to initial values.
From roma@zippy.cso.uiuc.edu Tue Aug 23 11:16:37 2005 -0500 Status: RO X-Status: X-Keywords: Received: via dmail-4.1(10) for +INBOX; Tue, 23 Aug 2005 11:16:37 -0500 (CDT) Return-Path: roma@zippy.cso.uiuc.edu Received: from zippy.cso.uiuc.edu (loopback [127.0.0.1]) by zippy.cso.uiuc.edu (8.12.10/8.12.10) with ESMTP id j7NGGUCQ027294 for roma@uiuc.edu; Tue, 23 Aug 2005 11:16:30 -0500 Received: (from roma@localhost) by zippy.cso.uiuc.edu (8.12.10/8.12.10/Submit) id j7NGGUrc022618 for roma@uiuc.edu; Tue, 23 Aug 2005 11:16:30 -0500 Date: Tue, 23 Aug 2005 11:16:30 -0500 From: Jon Roma roma@zippy.cso.uiuc.edu Message-Id: 200508231616.j7NGGUrc022618@zippy.cso.uiuc.edu To: roma@uiuc.edu Subject: Test at Tue Aug 23 11:16:29 CDT 2005 X-Spam-Checker-Version: SpamAssassin 3.0.1 (2004-10-22) on zippy.cso.uiuc.edu X-Spam-Level: X-Spam-Status: No, score=-5.9 required=2.5 tests=ALL_TRUSTED,BAYES_00 autolearn=ham version=3.0.1 X-UID: 1
Tue Aug 23 11:16:29 CDT 2005
On Tue, 2005-08-23 at 11:50 -0500, Jon Roma wrote:
Has anyone else encountered weird date behavior with mbox-style folders?
Looks like timezone handling was completely broken in mboxes. I think most people haven't noticed it because most software nowadays don't seem to write timezones to the From-line. This patch should fix it? Index: src/lib-storage/index/mbox/mbox-from.c =================================================================== RCS file: /var/lib/cvs/dovecot/src/lib-storage/index/mbox/mbox-from.c,v retrieving revision 1.6 diff -u -r1.6 mbox-from.c --- src/lib-storage/index/mbox/mbox-from.c 19 Aug 2005 15:49:49 -0000 1.6 +++ src/lib-storage/index/mbox/mbox-from.c 28 Aug 2005 18:52:29 -0000 @@ -145,8 +145,8 @@ i_isdigit(msg[3]) && i_isdigit(msg[4]) && msg[5] == ' ') { /* numeric timezone, use it */ seen_timezone = TRUE; - timezone = (msg[1]-'0') * 1000 + (msg[2]-'0') * 100 + - (msg[3]-'0') * 10 +(msg[4]-'0'); + timezone = (msg[1]-'0') * 10*60*60 + (msg[2]-'0') * 60*60 + + (msg[3]-'0') * 10 + (msg[4]-'0'); if (msg[0] == '-') timezone = -timezone; msg += 6; } @@ -166,8 +166,8 @@ i_isdigit(msg[2]) && i_isdigit(msg[3]) && i_isdigit(msg[4]) && i_isdigit(msg[5])) { seen_timezone = TRUE; - timezone = (msg[2]-'0') * 1000 + (msg[3]-'0') * 100 + - (msg[4]-'0') * 10 +(msg[5]-'0'); + timezone = (msg[2]-'0') * 10*60*60 + (msg[3]-'0') * 60*60 + + (msg[4]-'0') * 10 + (msg[5]-'0'); if (msg[1] == '-') timezone = -timezone; } @@ -176,7 +176,7 @@ if (t == (time_t)-1) return -1; - t -= timezone * 60; + t -= timezone; *time_r = t; } else { /* assume local timezone */
Timo:
Sorry for not writing faster. The patch you supplied (below) applied
to 1.0-test80 and solves the weird date behavior. Thank you for your
fast and efficient service!
I'm already planning out my upgrade from UW to Dovecot. So far 1.0-test80
works fine for me, but once the patch gets applied, I'll probably grab
one of the more recent versions.
Thanks again.
Timo Sirainen
On Tue, 2005-08-23 at 11:50 -0500, Jon Roma wrote:
Has anyone else encountered weird date behavior with mbox-style folders?
Looks like timezone handling was completely broken in mboxes. I think most people haven't noticed it because most software nowadays don't seem to write timezones to the From-line.
This patch should fix it?
Index: src/lib-storage/index/mbox/mbox-from.c =================================================================== RCS file: /var/lib/cvs/dovecot/src/lib-storage/index/mbox/mbox-from.c,v retrieving revision 1.6 diff -u -r1.6 mbox-from.c --- src/lib-storage/index/mbox/mbox-from.c 19 Aug 2005 15:49:49 -0000 1.6 +++ src/lib-storage/index/mbox/mbox-from.c 28 Aug 2005 18:52:29 -0000 @@ -145,8 +145,8 @@ i_isdigit(msg[3]) && i_isdigit(msg[4]) && msg[5] == ' ') { /* numeric timezone, use it */ seen_timezone = TRUE; - timezone = (msg[1]-'0') * 1000 + (msg[2]-'0') * 100 + - (msg[3]-'0') * 10 +(msg[4]-'0'); + timezone = (msg[1]-'0') * 10*60*60 + (msg[2]-'0') * 60*60 + + (msg[3]-'0') * 10 + (msg[4]-'0'); if (msg[0] == '-') timezone = -timezone; msg += 6; } @@ -166,8 +166,8 @@ i_isdigit(msg[2]) && i_isdigit(msg[3]) && i_isdigit(msg[4]) && i_isdigit(msg[5])) { seen_timezone = TRUE; - timezone = (msg[2]-'0') * 1000 + (msg[3]-'0') * 100 + - (msg[4]-'0') * 10 +(msg[5]-'0'); + timezone = (msg[2]-'0') * 10*60*60 + (msg[3]-'0') * 60*60 + + (msg[4]-'0') * 10 + (msg[5]-'0'); if (msg[1] == '-') timezone = -timezone; }
@@ -176,7 +176,7 @@ if (t == (time_t)-1) return -1;
- t -= timezone * 60; + t -= timezone; *time_r = t; } else { /* assume local timezone */
participants (2)
-
Jon Roma
-
Timo Sirainen