[Dovecot] Date sort-order ignoring timezone, and a few Q's.
Timo, thanks for writing and releasing Dovecot. I've recently discovered it and have been very impressed at the speed and balance between flexibility and simplicity.
I had to make a packet-trace to be sure that encryption with OpenSSL was working correctly, since it all 'just worked' first time without complex configuration, which I don't think I've ever seen before ;-) LDAP too was quite straightforward (after I worked out where my virtual-domain schema differed from what was required).
One small glitch I have noticed is that when the client requests a list of messages sorted by date, the timezone is ignored.
For example, listed in descending order:
07/07/2003 00:01 +0300 (= 21:01 UT) 06/07/2003 23:15 +0100 (= 22:15 UT) 06/07/2003 19:03 +0100 (= 18:03 UT)
I think that this should instead be...
06/07/2003 23:15 +0100 (= 22:15 UT) 07/07/2003 00:01 +0300 (= 21:01 UT) 06/07/2003 19:03 +0100 (= 18:03 UT)
It's most noticable when the client (Mulberry in my case) is configured to display all times in the local timezone. (Mulberry has recently added custom flags which work perfectly with Dovecot - I was hoping this would be the case, and was one of the reasons I was looking for something to replace the version of Courier I previously had installed). Mulberry does like to have NAMESPACE otherwise it usually asks for folder locations on connecting, btw. No real problem, though.
A little tip: if anyone is interested in having their MDA write Maildir with CR+LF so that sendfile(2) can be used on all messages (not just those created by Dovecot), Exim offers this natively by adding the "use_crlf" option to the appendfile transport which seems to work fine with 0.99.10 (though not 0.99.9.1, maybe because of 2003-06-04 17:39 in Changelog).
It seems Dovecot is using the ,W= Maildir tag for some type of size instead of ,S= but I haven't traced far enough to work out what exactly W/virtual_size means, am I right in guessing that it might be 'wire size' i.e. message-size-with-CRLF-line-endings? (If so, use_crlf in exim.conf can be augmented with "maildir_tag = ,W=$message_size" to help keep overheads to a minimum).
And, one final question: for a dual-stack machine to listen on both IPv6 and IPv4 sockets, is the currently preferred option simply to start two instances of Dovecot with their own dovecot.conf containing different *_listen parameters?
--Stu
On Mon, 2003-07-07 at 03:51, Stuart Henderson wrote:
One small glitch I have noticed is that when the client requests a list of messages sorted by date, the timezone is ignored.
It's not ignored, I just recently "fixed" it. Now it's properly fixed :)
It seems Dovecot is using the ,W= Maildir tag for some type of size instead of ,S= but I haven't traced far enough to work out what exactly W/virtual_size means, am I right in guessing that it might be 'wire size' i.e. message-size-with-CRLF-line-endings?
Yes. S means the actual size of the file, W is the size with CRs.
And, one final question: for a dual-stack machine to listen on both IPv6 and IPv4 sockets, is the currently preferred option simply to start two instances of Dovecot with their own dovecot.conf containing different *_listen parameters?
Currently, yes. I'll probably add support for multiple listening sockets per server some day.
On Wed, Jul 09, 2003 at 05:55:42PM +0300, Timo Sirainen wrote:
On Mon, 2003-07-07 at 03:51, Stuart Henderson wrote:
One small glitch I have noticed is that when the client requests a list of messages sorted by date, the timezone is ignored.
It's not ignored, I just recently "fixed" it. Now it's properly fixed :)
What version is it fixed in?
(we're having the same issue... with mbox format, the timezone on the "From " line is ignored. currently running dovecot-0.99.10-rc4)
-mm-
On Wed, 2003-07-09 at 18:08, Mark E. Mallett wrote:
One small glitch I have noticed is that when the client requests a list of messages sorted by date, the timezone is ignored.
It's not ignored, I just recently "fixed" it. Now it's properly fixed :)
What version is it fixed in?
Only in CVS. I could put out 0.99.11-test1.
On Wed, Jul 09, 2003 at 06:19:11PM +0300, Timo Sirainen wrote:
On Wed, 2003-07-09 at 18:08, Mark E. Mallett wrote:
One small glitch I have noticed is that when the client requests a list of messages sorted by date, the timezone is ignored.
It's not ignored, I just recently "fixed" it. Now it's properly fixed :)
What version is it fixed in?
Only in CVS. I could put out 0.99.11-test1.
That would be great
On Wed, 2003-07-09 at 18:22, Mark E. Mallett wrote:
Only in CVS. I could put out 0.99.11-test1. That would be great
On Wed, Jul 09, 2003 at 07:00:41PM +0300, Timo Sirainen wrote:
On Wed, 2003-07-09 at 18:22, Mark E. Mallett wrote:
Only in CVS. I could put out 0.99.11-test1. That would be great
Hmm... making imap fails now:
../lib/liblib.a(randgen.o): In function ssl_last_error': /usr/local/src/system/dovecot-0.99.11-test1/src/lib/randgen.c:90: undefined refer ence to ERR_get_error'
/usr/local/src/system/dovecot-0.99.11-test1/src/lib/randgen.c:96: undefined refer
ence to ERR_error_string_n' ../lib/liblib.a(randgen.o): In function random_fill':
/usr/local/src/system/dovecot-0.99.11-test1/src/lib/randgen.c:102: undefined refe
rence to `RAND_bytes'
/usr/local/src/system/dovecot-0.99.10-rc4 still builds fine on the same system.
I diffed the make outputs and they pretty much differ only at that point. Haven't spent any time to track it down, figured I'd ask first..
mm
On Wed, 2003-07-09 at 21:59, Mark E. Mallett wrote:
../lib/liblib.a(randgen.o): In function
ssl_last_error': /usr/local/src/system/dovecot-0.99.11-test1/src/lib/randgen.c:90: undefined refer ence toERR_get_error'
It's because we want random number generator for imap/pop3 binaries nowadays. I didn't notice it since I have /dev/urandom. Fixed in -test2.
On Wed, Jul 09, 2003 at 10:30:57PM +0300, Timo Sirainen wrote:
On Wed, 2003-07-09 at 21:59, Mark E. Mallett wrote:
../lib/liblib.a(randgen.o): In function
ssl_last_error': /usr/local/src/system/dovecot-0.99.11-test1/src/lib/randgen.c:90: undefined refer ence toERR_get_error'It's because we want random number generator for imap/pop3 binaries nowadays. I didn't notice it since I have /dev/urandom. Fixed in -test2.
Hmm, I have /dev/urandom too--
does this also address the other two undefineds?
On Wed, 2003-07-09 at 22:32, Mark E. Mallett wrote:
../lib/liblib.a(randgen.o): In function
ssl_last_error': /usr/local/src/system/dovecot-0.99.11-test1/src/lib/randgen.c:90: undefined refer ence toERR_get_error'It's because we want random number generator for imap/pop3 binaries nowadays. I didn't notice it since I have /dev/urandom. Fixed in -test2.
Hmm, I have /dev/urandom too--
Oh. I guess "test -c /dev/urandom" isn't the proper way to check it then. It's not a character device with you? What OS? Maybe test -e would work.
does this also address the other two undefineds?
yes.
On Wed, Jul 09, 2003 at 10:34:52PM +0300, Timo Sirainen wrote:
On Wed, 2003-07-09 at 22:32, Mark E. Mallett wrote:
../lib/liblib.a(randgen.o): In function
ssl_last_error': /usr/local/src/system/dovecot-0.99.11-test1/src/lib/randgen.c:90: undefined refer ence toERR_get_error'It's because we want random number generator for imap/pop3 binaries nowadays. I didn't notice it since I have /dev/urandom. Fixed in -test2.
Hmm, I have /dev/urandom too--
Oh. I guess "test -c /dev/urandom" isn't the proper way to check it then. It's not a character device with you? What OS? Maybe test -e would work.
no, I'm running BSD/OS 4.2, where /dev/random is implemented as (ugh) a symbolic link to a named socket attached to a "randomd" program. I should have mentioned this, since I have had this issue with other configures too. Not sure you can generalize for that :-)
10# cd /dev 11# ll urandom lrwxr-xr-x 1 root wheel 19 Jul 5 22:25 urandom@ -> /var/run/random.md5
12# file /var/run/random.md5 /var/run/random.md5: socket
-mm-
Timo Sirainen <tss@iki.fi> writes:
Oh. I guess "test -c /dev/urandom" isn't the proper way to check it then. It's not a character device with you? What OS? Maybe test -e would work.
test -e isn't implemented in Solaris' /bin/sh, so it's not an option unless you're willing to add special treatment to figure if you need to launch /usr/xpg4/bin/sh or something.
-- Matthias Andree
On Thu, 2003-07-10 at 03:26, Matthias Andree wrote:
Timo Sirainen <tss@iki.fi> writes:
Oh. I guess "test -c /dev/urandom" isn't the proper way to check it then. It's not a character device with you? What OS? Maybe test -e would work.
test -e isn't implemented in Solaris' /bin/sh, so it's not an option unless you're willing to add special treatment to figure if you need to launch /usr/xpg4/bin/sh or something.
I knew there was something wrong with it :) Well, this should work then:
if test -c /dev/urandom -o -s /dev/urandom; then
Timo Sirainen <tss@iki.fi> writes:
On Thu, 2003-07-10 at 03:26, Matthias Andree wrote:
Timo Sirainen <tss@iki.fi> writes:
Oh. I guess "test -c /dev/urandom" isn't the proper way to check it then. It's not a character device with you? What OS? Maybe test -e would work.
test -e isn't implemented in Solaris' /bin/sh, so it's not an option unless you're willing to add special treatment to figure if you need to launch /usr/xpg4/bin/sh or something.
I knew there was something wrong with it :) Well, this should work then:
if test -c /dev/urandom -o -s /dev/urandom; then
Autoconf suggests using
if test -c /dev/urandom || test -s /dev/urandom ; then
Rationale: Quoting from "(autoconf.info)Limitations of Builtins":
| test' | [...] | | If you need to make multiple checks using test', combine them with
| the shell operators &&' and ||' instead of using the test' | operators -a' and -o'. On System V, the precedence of -a' and
| -o' is wrong relative to the unary operators; consequently, POSIX | does not specify them, so using them is nonportable. If you | combine &&' and ||' in the same statement, keep in mind that | they have equal precedence. | | You may use !' with test', but not with if': `test ! -r foo ||
| exit 1'.
-- Matthias Andree
Timo Sirainen <tss@iki.fi> writes:
Oh well, changed. I've never however seen missing -a or -o, even Solaris has it :)
If it's not standardized, don't use it unless you have fallbacks. Look at the errno fuss with glibc 2.3. The nonstandard "extern int errno;" no longer works. #include <errno.h> continues to work.
-- Matthias Andree
On Wed, 9 Jul 2003, Mark E. Mallett wrote:
On Wed, Jul 09, 2003 at 06:19:11PM +0300, Timo Sirainen wrote:
On Wed, 2003-07-09 at 18:08, Mark E. Mallett wrote:
One small glitch I have noticed is that when the client requests a list of messages sorted by date, the timezone is ignored.
It's not ignored, I just recently "fixed" it. Now it's properly fixed :)
What version is it fixed in?
Only in CVS. I could put out 0.99.11-test1.
That would be great
Better for me would be a patch against 0.99.10.
-- Charlie
On Wed, 2003-07-09 at 21:13, Charlie Brady wrote:
One small glitch I have noticed is that when the client requests a list of messages sorted by date, the timezone is ignored. Better for me would be a patch against 0.99.10.
Index: message-date.c =================================================================== RCS file: /home/cvs/dovecot/src/lib-mail/message-date.c,v retrieving revision 1.7 retrieving revision 1.8 diff -u -d -r1.7 -r1.8 --- message-date.c 22 Apr 2003 17:47:58 -0000 1.7 +++ message-date.c 9 Jul 2003 14:51:56 -0000 1.8 @@ -208,7 +208,7 @@ if (*time == (time_t)-1) return FALSE; - *time += *timezone_offset * 60; + *time -= *timezone_offset * 60; return TRUE; } Index: index-search.c =================================================================== RCS file: /home/cvs/dovecot/src/lib-storage/index/index-search.c,v retrieving revision 1.74 retrieving revision 1.75 diff -u -d -r1.74 -r1.75 --- index-search.c 15 Jun 2003 03:42:29 -0000 1.74 +++ index-search.c 9 Jul 2003 14:51:57 -0000 1.75 @@ -297,7 +297,7 @@ if (!message_date_parse(sent_value, sent_value_len, &sent_time, &timezone_offset)) return 0; - sent_time -= timezone_offset * 60; + sent_time += timezone_offset * 60; switch (type) { case SEARCH_SENTBEFORE:
On 9 Jul 2003, Timo Sirainen wrote:
On Wed, 2003-07-09 at 21:13, Charlie Brady wrote:
Better for me would be a patch against 0.99.10.
Index: message-date.c
RCS file: /home/cvs/dovecot/src/lib-mail/message-date.c,v
Thanks, works for me (although I needed to mangle paths in the diff before patch could use it).
-- Charlie
On Wed, Jul 09, 2003 at 05:55:42PM +0300, Timo Sirainen wrote:
On Mon, 2003-07-07 at 03:51, Stuart Henderson wrote:
One small glitch I have noticed is that when the client requests a list of messages sorted by date, the timezone is ignored.
It's not ignored, I just recently "fixed" it. Now it's properly fixed :)
test2 does not fix the internaldate issue here, I'm sorry to say.
This only recently started happening, I think as of rc4 or thereabouts. Up until that time, the internaldate was being reported properly. Now it's not. (This is for mbox formats.)
More details..
It appears that the internaldate is being taken from the "From " line (aka the "From_" line). Traditionally local delivery agents would write that line something like:
From mem@mv.mv.com Thu Jul 10 16:46:11 2003with no timezone info at all. However different local delivery agents would use various timezones, some using GMT or UT, and some using whatever local time was in effect. Plus, if you take a saved mbox with you to another timezone, all those timestamps are wrong. At some point in the past, we made the local delivery agents use timezone 0 and append the keyword UTC at the end, like this:
From mem@mv.mv.com Thu Jul 10 20:46:11 2003 UTC(I realize that standard mbox "From_" date is written in local timezone with no timezone qualifier, but I really dislike that amount of ambiguity so I did what I did.)
Up until recently dovecot was returning the correct internaldate. Recently it has started returning the time string as if it were the local time, and ignoring the timezone.
I suspected that you were being more rigorous about ignoring known obsolete forms including keyword timezones, so I changed to:
From mem@mv.mv.com Thu Jul 10 20:46:11 2003 +0000with no change in behaviour. Apparently you are being more rigorous about enforcing standard mbox "From_" format?
Would it hurt to parse timezone information on the "From_" line if it's there?
mm
On Friday, Jul 11, 2003, at 00:52 Europe/Helsinki, Mark E. Mallett wrote:
Up until recently dovecot was returning the correct internaldate. Recently it has started returning the time string as if it were the local time, and ignoring the timezone.
I doubt it has ever even tried to use the timezone in From-lines. If it worked with you before, it was probably because you had the dates stored in "correct" timezone.
Would it hurt to parse timezone information on the "From_" line if it's there?
I guess I could do it. At least the -0000 or +0000 format. Parsing named timezones would probably be too much trouble.
On Fri, Jul 11, 2003 at 02:45:11AM +0300, Timo Sirainen wrote:
On Friday, Jul 11, 2003, at 00:52 Europe/Helsinki, Mark E. Mallett wrote:
Up until recently dovecot was returning the correct internaldate. Recently it has started returning the time string as if it were the local time, and ignoring the timezone.
I doubt it has ever even tried to use the timezone in From-lines. If it worked with you before, it was probably because you had the dates stored in "correct" timezone.
Nope-- no change there for a long, long time. It suddenly started happening after a particular dovecot install. I suspected that it had something to do with the change you made to start caching dates if they were referenced (am I remembering that right?) Perhaps without caching you simply took the literal string, but with caching you interpreted the date?
Would it hurt to parse timezone information on the "From_" line if it's there?
I guess I could do it. At least the -0000 or +0000 format. Parsing named timezones would probably be too much trouble.
That would be great, even though I'm probably the only one who wants it.
mm
On Friday, Jul 11, 2003, at 02:49 Europe/Helsinki, Mark E. Mallett wrote:
I doubt it has ever even tried to use the timezone in From-lines. If it worked with you before, it was probably because you had the dates stored in "correct" timezone.
Nope-- no change there for a long, long time. It suddenly started happening after a particular dovecot install. I suspected that it had something to do with the change you made to start caching dates if they were referenced (am I remembering that right?) Perhaps without caching you simply took the literal string, but with caching you interpreted the date?
No, they're always parsed. And I've always ignored timezones in it. Looks like I added support for ".. Jul 11 EET 2003" format where it just skips the timezone, before that it would treat such From lines completely invalid. Maybe that's what happened with you? If Dovecot thinks that the timestamp is invalid, it uses current time (parsing time) as internaldate.
On Fri, Jul 11, 2003 at 03:08:56AM +0300, Timo Sirainen wrote:
No, they're always parsed. And I've always ignored timezones in it. Looks like I added support for ".. Jul 11 EET 2003" format where it just skips the timezone, before that it would treat such From lines completely invalid. Maybe that's what happened with you? If Dovecot thinks that the timestamp is invalid, it uses current time (parsing time) as internaldate.
Ah, probably.
participants (5)
-
Charlie Brady
-
Mark E. Mallett
-
Matthias Andree
-
Stuart Henderson
-
Timo Sirainen