[Dovecot] 1.0alpha4: pop3_reuse_xuidl patch
hi all, this is needed to make the pop3_reuse_xuidl option work in 1.0alpha4. I am using maildir and I noticed that for each message which doesn't have an indexed X-UIDL header, the entire message is read from disk. as this is reading only headers, it is probably reasonable to stop reading at the end of the headers. it would also be neat if dovecot could be configured to only try reading X-UIDL headers from messages if the mailbox indexes don't exist, since I guess it would be common to want to preserve existing X-UIDLs but not bother for new messages (perhaps some mtime comparison, or something). opinions? I haven't looked at the source enough yet to be able to do this, but these two optimizations would vastly reduce the amount of disk IO as a result of using this option. grant. [apologies if this shows up twice, I originally sent it from an address which isn't subscribed, and it got held] --- src/pop3/commands.c.orig 2005-10-17 02:21:39.000000000 +1000 +++ src/pop3/commands.c 2005-11-09 15:26:15.000000000 +1100 @@ -567,6 +567,8 @@ mail_get_first_header(ctx->mail, "X-UIDL"); if (uidl == NULL) var_expand(str, uidl_format, tab); + else + str_append(str, uidl); ret = client_send_line(client, "%s", str_c(str)); t_pop();
On Fri, 2005-11-11 at 16:38 +1100, grant beattie wrote:
this is needed to make the pop3_reuse_xuidl option work in 1.0alpha4.
Committed to CVS finally.
I am using maildir and I noticed that for each message which doesn't have an indexed X-UIDL header, the entire message is read from disk. as this is reading only headers, it is probably reasonable to stop reading at the end of the headers.
I don't think it reads the message body because of that?
If message size isn't yet in cache file, the body needs to be read to get the correct size with CR+LF linefeeds.
it would also be neat if dovecot could be configured to only try reading X-UIDL headers from messages if the mailbox indexes don't exist, since I guess it would be common to want to preserve existing X-UIDLs but not bother for new messages (perhaps some mtime comparison, or something).
opinions? I haven't looked at the source enough yet to be able to do this, but these two optimizations would vastly reduce the amount of disk IO as a result of using this option.
Well, I originally thought the setting would only be used with mboxes since it's mostly for backwards compatibility, and with mboxes reading X-UIDL always is much cheaper. What maildir pop3 servers exist that don't use the filename or IMAP UIDs in some way for the UIDL?
Actually since the message size needs to be read anyway (unless you can get it to be written as W=1234 in maildir file), the X-UIDL lookup shouldn't cost anything since it's written to cache file immediately. Except I'm not sure if Dovecot's caching code currently is smart enough to save the X-UIDL header while saving message sizes..
On Tue, Dec 06, 2005 at 07:45:55PM +0200, Timo Sirainen wrote:
On Fri, 2005-11-11 at 16:38 +1100, grant beattie wrote:
this is needed to make the pop3_reuse_xuidl option work in 1.0alpha4.
Committed to CVS finally.
cool, thanks :)
I am using maildir and I noticed that for each message which doesn't have an indexed X-UIDL header, the entire message is read from disk. as this is reading only headers, it is probably reasonable to stop reading at the end of the headers.
I don't think it reads the message body because of that?
If message size isn't yet in cache file, the body needs to be read to get the correct size with CR+LF linefeeds.
ah, that makes sense then, I guess...
opinions? I haven't looked at the source enough yet to be able to do this, but these two optimizations would vastly reduce the amount of disk IO as a result of using this option.
Well, I originally thought the setting would only be used with mboxes since it's mostly for backwards compatibility, and with mboxes reading X-UIDL always is much cheaper. What maildir pop3 servers exist that don't use the filename or IMAP UIDs in some way for the UIDL?
I don't know if any exist, but in my case the mboxes were from a qpopper installation, and they were converted to maildir when I migrated to dovecot.
overall it has been a very pleasant experience :)
Actually since the message size needs to be read anyway (unless you can get it to be written as W=1234 in maildir file), the X-UIDL lookup shouldn't cost anything since it's written to cache file immediately. Except I'm not sure if Dovecot's caching code currently is smart enough to save the X-UIDL header while saving message sizes..
ooh, so W=<size> will optimize away having to read the whole message? that's great! I am currently using S=<size> (for Exim quota calculation purposes), but I will switch to W= if it will reduce disk IO :)
grant.
On Wed, 2005-12-07 at 07:35 +1100, grant beattie wrote:
Actually since the message size needs to be read anyway (unless you can get it to be written as W=1234 in maildir file), the X-UIDL lookup shouldn't cost anything since it's written to cache file immediately. Except I'm not sure if Dovecot's caching code currently is smart enough to save the X-UIDL header while saving message sizes..
ooh, so W=<size> will optimize away having to read the whole message? that's great! I am currently using S=<size> (for Exim quota calculation purposes), but I will switch to W= if it will reduce disk IO :)
Yes, but you shouldn't just fake W= with wrong size or things can break.
S= is used for the file's actual size. W= is used for the "virtual size", the size that the message would have with CR+LF linefeeds. So you could have both.
On Tue, Dec 06, 2005 at 10:50:21PM +0200, Timo Sirainen wrote:
On Wed, 2005-12-07 at 07:35 +1100, grant beattie wrote:
Actually since the message size needs to be read anyway (unless you can get it to be written as W=1234 in maildir file), the X-UIDL lookup shouldn't cost anything since it's written to cache file immediately. Except I'm not sure if Dovecot's caching code currently is smart enough to save the X-UIDL header while saving message sizes..
ooh, so W=<size> will optimize away having to read the whole message? that's great! I am currently using S=<size> (for Exim quota calculation purposes), but I will switch to W= if it will reduce disk IO :)
Yes, but you shouldn't just fake W= with wrong size or things can break.
S= is used for the file's actual size. W= is used for the "virtual size", the size that the message would have with CR+LF linefeeds. So you could have both.
does Dovecot use S= for anything? and are these filename tokens configurable and/or documented? I couldn't find a mention in the config file.
I should be able to get Exim to do anything I want with Maildir filenames, so I'll look into this later today. thanks for the tip :)
grant.
On Wed, 2005-12-07 at 07:52 +1100, grant beattie wrote:
Yes, but you shouldn't just fake W= with wrong size or things can break.
S= is used for the file's actual size. W= is used for the "virtual size", the size that the message would have with CR+LF linefeeds. So you could have both.
does Dovecot use S= for anything?
Dovecot's quota plugin at least can use it when it needs to find out message's size.
and are these filename tokens configurable and/or documented? I couldn't find a mention in the config file.
Not really. S= is from Maildir++ standard and I think W= is also used by Courier.
participants (2)
-
grant beattie
-
Timo Sirainen