[Dovecot] SORT using cached filesystem modtimes
Timo or anyone,
This may be a bug or maybe a design decision. Either way I am curious to hear comments.
Why do the dovecot-0.99.14 server's .imap.index files contain a cached copy of each message's date and time as supplied by the filesystem, as opposed to the date and time found in each message header?
When testing various imap clients (eg. tbird-1.0.2, evolution-2.0, outlook2003, squirrelmail-1.4.4) I was intrigued by the behaviour of message date sorting. I toggled on/off the use of server side SORTing in some imap clients, then started forcing rebuilds of .map.index files, and eventually came to the above conclusion.
For normal dovecot server side processing, this condition (if I am correct) may not be a problem, but if someone was to deposit a bunch of maildir files firstly into an imap account (bypassing dovecot), and if this person (eg. me) did not preserve the filesystem/inode file attributes. Then dovecot appears to cache the "wrong" data. What happens then is that some imap clients can display wrong date+times in the message lists, and some imap clients find that server side SORTing does not work.
Keen to hear thoughts/opinions.
-Emmett
Emmett Lazich wrote:
Timo or anyone,
This may be a bug or maybe a design decision. Either way I am curious to hear comments.
Why do the dovecot-0.99.14 server's .imap.index files contain a cached copy of each message's date and time as supplied by the filesystem, as opposed to the date and time found in each message header?
When testing various imap clients (eg. tbird-1.0.2, evolution-2.0, outlook2003, squirrelmail-1.4.4) I was intrigued by the behaviour of message date sorting. I toggled on/off the use of server side SORTing in some imap clients, then started forcing rebuilds of .map.index files, and eventually came to the above conclusion.
For normal dovecot server side processing, this condition (if I am correct) may not be a problem, but if someone was to deposit a bunch of maildir files firstly into an imap account (bypassing dovecot), and if this person (eg. me) did not preserve the filesystem/inode file attributes. Then dovecot appears to cache the "wrong" data. What happens then is that some imap clients can display wrong date+times in the message lists, and some imap clients find that server side SORTing does not work.
Keen to hear thoughts/opinions.
Emmet,
You hit the nail on the head for me.. I'm using the latest 1.0-test67 (IIRC) build and also saw this with all of the other older versions as well.. I find it rather odd as well -- mostly for my SPAM directory which is where dspam dumps my spam.. About 10-20% of the time, much of the spam has a date (according to the Thunderbird email client) of 12/31/69 at 4:00pm.. If I find one of the offending files though, I can't find where it got the 1969 date.. I also occasionally get this behavior in my inbox as well..
If you've got any ideas on how to ditch this behavior without disabling indexes, I'd be interested to hear how to do it.. I've got some rather large Maildir folders (for mailing lists) with >10,000 messages in them and would prefer to keep the indexes if possible..
-- Rick
On Fri, 2005-04-15 at 15:31 +1000, Emmett Lazich wrote:
Timo or anyone,
This may be a bug or maybe a design decision. Either way I am curious to hear comments.
Why do the dovecot-0.99.14 server's .imap.index files contain a cached copy of each message's date and time as supplied by the filesystem, as opposed to the date and time found in each message header?
There are two dates with IMAP. INTERNALDATE is one and Date-header of the message is another. The filesystem timestamp that you see cached in .imap.index.data file is the INTERNALDATE. The message's Date-header is also cached in the beginning of ENVELOPE data.
When testing various imap clients (eg. tbird-1.0.2, evolution-2.0, outlook2003, squirrelmail-1.4.4) I was intrigued by the behaviour of message date sorting. I toggled on/off the use of server side SORTing in some imap clients, then started forcing rebuilds of .map.index files, and eventually came to the above conclusion.
Server side sorting supports sorting based on both INTERNALDATE ("ARRIVAL") and Date-header ("DATE"). It's client's choice which one it wants to use.
For normal dovecot server side processing, this condition (if I am correct) may not be a problem, but if someone was to deposit a bunch of maildir files firstly into an imap account (bypassing dovecot), and if this person (eg. me) did not preserve the filesystem/inode file attributes. Then dovecot appears to cache the "wrong" data. What happens then is that some imap clients can display wrong date+times in the message lists, and some imap clients find that server side SORTing does not work.
It all works correctly from Dovecot's point of view, because the messages arrived in the mailbox at that time..
Thanks Timo. Very interesting and very good by you.
My only followup thought is to ask: Is it possible for an IMAP client to request a server side SORT, but not specify explicitly which date to use, and thus rely on a server side choice of the SORT key? In which case could Dovecot use the Date-header as the default SORT key?
On Mon, 2005-04-18 at 02:35, Timo Sirainen wrote:
On Fri, 2005-04-15 at 15:31 +1000, Emmett Lazich wrote:
Timo or anyone,
This may be a bug or maybe a design decision. Either way I am curious to hear comments.
Why do the dovecot-0.99.14 server's .imap.index files contain a cached copy of each message's date and time as supplied by the filesystem, as opposed to the date and time found in each message header?
There are two dates with IMAP. INTERNALDATE is one and Date-header of the message is another. The filesystem timestamp that you see cached in .imap.index.data file is the INTERNALDATE. The message's Date-header is also cached in the beginning of ENVELOPE data.
When testing various imap clients (eg. tbird-1.0.2, evolution-2.0, outlook2003, squirrelmail-1.4.4) I was intrigued by the behaviour of message date sorting. I toggled on/off the use of server side SORTing in some imap clients, then started forcing rebuilds of .map.index files, and eventually came to the above conclusion.
Server side sorting supports sorting based on both INTERNALDATE ("ARRIVAL") and Date-header ("DATE"). It's client's choice which one it wants to use.
For normal dovecot server side processing, this condition (if I am correct) may not be a problem, but if someone was to deposit a bunch of maildir files firstly into an imap account (bypassing dovecot), and if this person (eg. me) did not preserve the filesystem/inode file attributes. Then dovecot appears to cache the "wrong" data. What happens then is that some imap clients can display wrong date+times in the message lists, and some imap clients find that server side SORTing does not work.
It all works correctly from Dovecot's point of view, because the messages arrived in the mailbox at that time..
participants (3)
-
Emmett Lazich
-
Rick Flower
-
Timo Sirainen