Re: [Dovecot] changing INTERNALDATE or similar
Looking at the IMAP4rev1 protocol (RFC 2060), we have the following definition:
INTERNALDATE The internal date of the message. 2.3.3. Internal Date Message Attribute
The internal date and time of the message on the server. This is not the date and time in the [RFC-822] header, but rather a date and time which reflects when the message was received. In the case of messages delivered via [SMTP], this SHOULD be the date and time of final delivery of the message as defined by [SMTP]. In the case of messages delivered by the IMAP4rev1 COPY command, this SHOULD be the internal date and time of the source message. In the case of messages delivered by the IMAP4rev1 APPEND command, this SHOULD be the date and time as specified in the APPEND command description. All other cases are implementation defined.
How could any implementation of this protocol possibly use a file system time stamp to represent that important piece of meta-data, no matter where the file lives? It seems totally reasonable that this is what Outlook uses for the Received date (and I rarely defend Microsoft).
This seems like a real design flaw in the dovecot implementation. I am fairly new to dovecot (and like many aspects of it over uw-imap), but having to really be careful with my mail store's mtimes borders on the absurd. I realize it is "implementation defined", but the intent of the definition surely does not refer to file system time stamps. Any chance this can be reconsidered? Is this an actual dovecot issue or a more general Maildir issue?
On Fri, 2008-07-18 at 02:00 -0700, dovecot-request@dovecot.org wrote:
Message: 4 Date: Thu, 17 Jul 2008 23:12:55 +0300 From: Timo Sirainen <tss@iki.fi> Subject: Re: [Dovecot] changing INTERNALDATE or similar To: richs@whidbey.net Cc: Dovecot Mailing List <dovecot@dovecot.org> Message-ID: <1216325575.31765.349.camel@hurina> Content-Type: text/plain; charset="utf-8"
On Thu, 2008-07-17 at 13:04 -0700, richs@whidbey.net wrote:
I believe it's still true that any mail files that are IMAP STORE'd (uploaded)
You mean IMAP APPENDed.
to Dovecot by a client will have a current mtime, rather than the "Date" in the message. That means might need to compare mtime's to "Date" headers for new files that appear too.
This is only because the client doesn't specify the date to APPEND command.
Here we just patched Dovecot to treat "INTERNALDATE" the same as the "Date" header. Constantly verifying mtime's became too much of a chore for those clients that rely on it (although this technically is against the RFC).
Would probably have been better to set the default INTERNALDATE based on the Date: header in APPEND command. Should be pretty easy to do with Maildir I think.
On Jul 18, 2008, at 6:33 PM, Karl Rudnick wrote:
How could any implementation of this protocol possibly use a file
system time stamp to represent that important piece of meta-data, no matter where the file lives? It seems totally reasonable that
this is what Outlook uses for the Received date (and I rarely defend
Microsoft).This seems like a real design flaw in the dovecot implementation. I
am fairly new to dovecot (and like many aspects of it over uw-imap), but having to really be careful with my mail store's mtimes borders on the
absurd. I realize it is "implementation defined", but the intent of the
definition surely does not refer to file system time stamps. Any chance this can be reconsidered? Is this an actual dovecot issue or a more general
Maildir issue?
Why does it matter where the timestamp lives? No matter how it was
stored, you would have had the exact same problem because your client
told Dovecot to use the current timestamp when saving the messages.
And why would keeping the INTERNALDATE in mtime be bad? It only
changes if you write to the file. And existing mails must not be
modified or you'll get other problems as well.
Why does it matter where the timestamp lives? No matter how it was stored, you would have had the exact same problem because your client told Dovecot to use the current timestamp when saving the messages.
And why would keeping the INTERNALDATE in mtime be bad? It only changes if you write to the file. And existing mails must not be modified or you'll get other problems as well.
I guess that with the zlib plugin this date is extracted from the gzip metadata, right? I didn't preserve mtimes of gzipped mails...
-- Mateusz
On Jul 18, 2008, at 7:19 PM, Mateusz Kijowski wrote:
Why does it matter where the timestamp lives? No matter how it was stored, you would have had the exact same problem because your client told Dovecot to use the current timestamp when saving the messages.
And why would keeping the INTERNALDATE in mtime be bad? It only changes if you write to the file. And existing mails must not be modified or you'll get other problems as well.
I guess that with the zlib plugin this date is extracted from the gzip metadata, right? I didn't preserve mtimes of gzipped mails...
I hadn't thought of that before. Yes, I should mention this in the
wiki page that the mtime should be preserved.
On Fri, 2008-07-18 at 08:52 -0700, Timo Sirainen wrote:
Why does it matter where the timestamp lives? No matter how it was
stored, you would have had the exact same problem because your client
told Dovecot to use the current timestamp when saving the messages.And why would keeping the INTERNALDATE in mtime be bad? It only
changes if you write to the file. And existing mails must not be
modified or you'll get other problems as well.
Actually, it wasn't the client (Outlook) that told Dovecot. I rarely save to my IMAP server from Outlook - only Thunderbird and Evolution. I think it was a result of my original migration from mbox (uw-imap) to Maildir (uw-imap). I did this by having both IMAP servers (uw and dovecot) accounts alive simultaneously in evolution and let evolution copy from the uw-account to the dovecot-imap account. It could be evolution's fault, but I thought it was doing an IMAP4 copy, in which case the INTERNALDATE should have been preserved according to the spec, but maybe evolution was just doing its own thing. So, before the migration, there was not even the concept of a timestamp on the uw-imap store for each mail.
Unfortunately, dates can change unexpectedly - not just when you modify the file, as I move data around between machines as they become obsolete and the wrong settings in the move can obliterate those mtimes - we humans do make mistakes. Nevertheless, I am happy with my current situation and at least understand what's going on.
THANKS, all, for your explanations and suggestions
On Jul 18, 2008, at 7:20 PM, Karl Rudnick wrote:
On Fri, 2008-07-18 at 08:52 -0700, Timo Sirainen wrote:
Why does it matter where the timestamp lives? No matter how it was stored, you would have had the exact same problem because your client told Dovecot to use the current timestamp when saving the messages.
And why would keeping the INTERNALDATE in mtime be bad? It only changes if you write to the file. And existing mails must not be modified or you'll get other problems as well.
Actually, it wasn't the client (Outlook) that told Dovecot. I rarely save to my IMAP server from Outlook - only Thunderbird and
Evolution. I think it was a result of my original migration from mbox (uw-imap) to Maildir (uw-imap). I did this by having both IMAP servers (uw and dovecot) accounts alive simultaneously in evolution and let evolution copy from the uw-account to the dovecot-imap account. It could be evolution's fault, but I thought it was doing an IMAP4 copy, in which case the INTERNALDATE should have been preserved according to the
spec, but maybe evolution was just doing its own thing.
You had two servers. There's no trans-server COPY command, so the
client has to fetch the messages from one server and save the messages
to another using APPEND command, which takes a parameter to specify
the INTERNALDATE.
So, before the migration, there was not even the concept of a timestamp on the uw- imap store for each mail.
With mboxes it's stored in the From-line for each message.
Unfortunately, dates can change unexpectedly - not just when you
modify the file, as I move data around between machines as they become
obsolete and the wrong settings in the move can obliterate those mtimes - we humans do make mistakes. Nevertheless, I am happy with my current situation and at least understand what's going on.
Perhaps if you're not careful when copying. :) Anyway there isn't
really any other good place to store the INTERNALDATE than mtime.
Perhaps the timestamp in file name could be used, although maildir
filenames aren't required to start with the timestamp so it should
fallback to mtime then anyway.
participants (3)
-
Karl Rudnick
-
Mateusz Kijowski
-
Timo Sirainen