[Dovecot] Indexing Performance Question (was tpop3d vs dovecot)
Since posting the previous thread we've setup a new system (Opteron 2.0ghz, 1gb ram, Fedora 6) for testing. I am hoping somebody very familiar with the indexing portion of dovecot can shed some light on this for me.
After much testing, I've come to one primary conclusion that dovecot is possibly unnecessarily scanning or reading files within the maildir directories. Take a mailbox for instance (my test one has 71570 emails in the cur/ folder so the process runs long enough to get data of what it's doing). Follow these steps:
- First pop3 connection. With no index files whatsoever, it takes nearly a full 10 minutes at 100% io load to scan the 71570 files (748M) directory and build the index files from scratch which look like this: -rw------- 1 postfix postdrop 839K Feb 1 00:43 dovecot.index -rw------- 1 postfix postdrop 1.9M Feb 1 00:57 dovecot.index.cache -rw------- 1 postfix postdrop 560K Feb 1 00:43 dovecot.index.log -rw------- 1 postfix postdrop 3.1M Feb 1 00:43 dovecot-uidlist
While it's building index i see a lot of these fly by in lsof pop3 2023 postfix 8r REG 8,3 14425 285947 /var/spool/mail/test.visp.net/natetest/cur/1128488047.V902I7d42e.s3.visp.net:2,
No problem here, the disk can only go as fast as the disk can go and thats a *lot* of mail. Although, one curious thing here, it appears that dovecot is reading or at least scanning the entire contents of the files within the cur/ directory to generate an index. I assume this by watching my disk cache go from 100MB to 600MB. Is it necessary to do that or only read the top X lines of each file, is that even possible?
- Second connection. Near instantaneous results from a login and LIST. Cannot perform an lsof fast enough to see what files it's opening. This step can be repeated with nearly the same results.
*rebooted machine to clear all cache*
- Third connection. It takes ~40 seconds to display a LIST after login. The index files are untouched; however, during the 40 seconds I'm waiting, I can catch a few of these in lsof. pop3 1971 postfix 10r REG 8,3 4196 205963 /var/spool/mail/test.visp.net/natetest/cur/1157863512.V902I1f3867.s3.visp.net:2,S
This tells me even though there is no new email, it is re-scanning the cur/ directory somewhat. It doesn't take near as long as the initial index build, but certainly takes longer than it would for the system to read any or all of the listed index files above. Why would dovecot need to even enter cur/ if the index file is up to date? Is it verifying that the index file is up to date by re-reading some or all of the many files in cur/? If so, could it not get the info it needs by simply doing the equivalent of a ls, which would take considerably less time?
The disk cache is only filled up an additional 30meg vs an additional 500meg from connection 1, so it's certainly not reading as much data. (also makes me wonder why the initial scan/index build could not use this lesser time consuming function if thats what it's doing).
Additional information: Running strace on the initial indexing looks like this...
open("/var/spool/mail/test.visp.net/natetest/cur/1115929738.V902I67c13.test.visp.net:2,",
O_RDONLY) = 8
fstat(8, {st_mode=S_IFREG|0600, st_size=4667, ...}) = 0
pread(8, "Return-Path:
Running strace on the 3rd connection looks like this...
open("/var/spool/mail/test.visp.net/natetest/cur/1114490671.V902I6011f.test.visp.net:2,",
O_RDONLY) = 10
fstat(10, {st_mode=S_IFREG|0600, st_size=5367, ...}) = 0
pread(10, "Return-Path:
Thank you for the help!
- Nate
Running strace on the 3rd connection looks like this...
open("/var/spool/mail/test.visp.net/natetest/cur/1114490671.V902I6011f.test.visp.net:2,",
O_RDONLY) = 10
fstat(10, {st_mode=S_IFREG|0600, st_size=5367, ...}) = 0
pread(10, "Return-Path: Thank you for the help!
10 minutes to read 748Mb? Kinda slow, isn't it ? For me reading 2Gb mailbox (not maildir) takes MUCH less time than that (around 20 sec). It should be within 40secs to read the whole maildir.
FiL
Nate wrote:
- First pop3 connection. With no index files whatsoever, it takes nearly a full 10 minutes at 100% io load to scan the 71570 files (748M) directory and build the index files from scratch which look like this:
At 10:50 PM 1/31/2007, FiL wrote:
10 minutes to read 748Mb? Kinda slow, isn't it ? For me reading 2Gb mailbox (not maildir) takes MUCH less time than that (around 20 sec). It should be within 40secs to read the whole maildir.
I whole-heartedly agree, that is slow; however, not necessarily any problem with dovecot. More a filesystem issue. Because it's not 1 large synchronous file, the disk has to seek like crazy to read each file. Here's some testing to show it's normal:
I looped through all the files and cat them to /dev/null. This is independent of dovecot. This is reading the entire contents of every file to /dev/null. I don't know why or if dovecot would have to do that to build an index, should be able to get away with the first 100 lines max I'd think.
[root@test cur]# time for x in ls ; do cat $x > /dev/null ; done
real 15m28.568s
user 0m20.993s
sys 1m27.609s
Now if I copy the contents of cur/ to another directory (on an
otherwise idle machine), so it's all written synchronously on the
disk, the results change dramatically. (a reboot is performed to
clear all cache after the copy)
[root@test testspool]# time for x in ls ; do cat $x > /dev/null ; done
real 2m15.945s
user 0m20.493s
sys 1m10.796s
Finally, to simulate a mbox like you mention I tar all the files to 1 large file, reboot, and cat the new tarfile to /dev/null. [root@test natetest]# time cat cur.tar >/dev/null real 0m15.162s user 0m0.044s sys 0m1.024s
What's sad with these tests is it takes 2m15s to read all the data in 71570 individual files, and only 15s to do it in 1 file. Granted there's a bit of overhead from the shell script looping, but not *that* much! I wonder how these tests would go on a different file system. I'm currently using reiserfs.
- Nate
Too bad. I was almost ready to start migrating my mbox-based setup to Maildir... One of the most important reasons for that - to speed things up and lower disk IO. So now I just converted one 3Gb mailbox (88K mails) to Maildir and tried accessing it via pop3...
# telnet enders2 110 Trying 127.0.0.1... Connected to localhost.localdomain (127.0.0.1). Escape character is '^]'. +OK Dovecot ready. user filonovd +OK pass XXXXXX +OK Logged in. list < here comes looong wait -ERR Disconnected for inactivity. Connection closed by foreign host.
There is no index for that Maildir. But dovecot can't even make one in the first place. It just times out. It's really scares me. That means, that for pop3 users Maildir is so much slower, than mbox?
Please, somebody tell me this is not true.
FiL
Nate wrote:
I whole-heartedly agree, that is slow; however, not necessarily any problem with dovecot. More a filesystem issue. Because it's not 1 large synchronous file, the disk has to seek like crazy to read each file. Here's some testing to show it's normal:
... some test data skipped
What's sad with these tests is it takes 2m15s to read all the data in 71570 individual files, and only 15s to do it in 1 file. Granted there's a bit of overhead from the shell script looping, but not *that* much! I wonder how these tests would go on a different file system. I'm currently using reiserfs.
- Nate
At 08:57 PM 2/1/2007, FiL wrote:
Too bad. I was almost ready to start migrating my mbox-based setup to Maildir... One of the most important reasons for that - to speed things up and lower disk IO. So now I just converted one 3Gb mailbox (88K mails) to Maildir and tried accessing it via pop3...
# telnet enders2 110 Trying 127.0.0.1... Connected to localhost.localdomain (127.0.0.1). Escape character is '^]'. +OK Dovecot ready. user filonovd +OK pass XXXXXX +OK Logged in. list < here comes looong wait -ERR Disconnected for inactivity. Connection closed by foreign host.
There is no index for that Maildir. But dovecot can't even make one in the first place. It just times out. It's really scares me. That means, that for pop3 users Maildir is so much slower, than mbox?
Please, somebody tell me this is not true.
As of rc21 which fixes a major indexing issue (the start of this thread), this shouldn't be a concern for you anytime except the very first time a mailbox is indexed. From there on out it would only take a few milliseconds to index the new files which arrive. 88k messages must mean it's years worth of email (or tons of spam).
To do the first index you could increase the idle timeouts in dovecot (have to do it in the src/pop3/client.c) file as it's not a config option that i'm aware of. I just modified both the CLIENT_OUTPUT_TIMEOUT and CLIENT_IDLE_TIMEOUT to 60*60.
Another option would be to defrag that 88k mailspool by copying it somewhere else, then copying it back so the operation happens faster, although personally I'd just change the idle timeouts and grin and bear it the first time. maildir has so many advantages over mbox I would highly suggest it!
- Nate
On Fri, 2007-02-02 at 12:05 -0800, Nate wrote:
list < here comes looong wait -ERR Disconnected for inactivity. Connection closed by foreign host.
There is no index for that Maildir. But dovecot can't even make one in the first place. It just times out. It's really scares me. That means, that for pop3 users Maildir is so much slower, than mbox? .. To do the first index you could increase the idle timeouts in dovecot (have to do it in the src/pop3/client.c) file as it's not a config option that i'm aware of. I just modified both the CLIENT_OUTPUT_TIMEOUT and CLIENT_IDLE_TIMEOUT to 60*60.
Hmm. The real problem here is that all the hard work done with caching the data is all discarded unless the client sent a QUIT command. I suppose I'll fix this so that it'll always get saved.
That would be great, Timo. This way it will recreate the cache after second (third, fourth..) try and use it after, so future access fill be fast. Very good idea. Thanks.
FiL
Timo Sirainen wrote:
Hmm. The real problem here is that all the hard work done with caching the data is all discarded unless the client sent a QUIT command. I suppose I'll fix this so that it'll always get saved.
I don't have plans to release rc22 soon, but you can apply that patch
directly:
http://dovecot.org/list/dovecot-cvs/2007-February/007602.html
On 5.2.2007, at 20.28, FiL @ Kpoxa wrote:
That would be great, Timo. This way it will recreate the cache
after second (third, fourth..) try and use it after, so future access fill be fast. Very good idea. Thanks.FiL
Timo Sirainen wrote:
Hmm. The real problem here is that all the hard work done with
caching the data is all discarded unless the client sent a QUIT command. I suppose I'll fix this so that it'll always get saved.
What's sad with these tests is it takes 2m15s to read all the data in 71570 individual files, and only 15s to do it in 1 file. Granted there's a bit of overhead from the shell script looping, but not *that* much! I wonder how these tests would go on a different file system. I'm currently using reiserfs.
actually the loop adds a lot of overhead: These commands were run right in a row so there is some speedup due to cacheing, So I repeted test a few times. Admittedly, this is a small maildir.
bash-2.05b# ls | wc 1009 1009 75500
bash-2.05b# time for x in ls ; do cat $x > /dev/null ; done
real 0m3.670s user 0m0.347s sys 0m1.508s bash-2.05b# time cat * > /dev/null
real 0m0.497s
user 0m0.015s
sys 0m0.071s
bash-2.05b# time for x in ls ; do cat $x > /dev/null ; done
real 0m2.237s user 0m0.356s sys 0m1.435s bash-2.05b# time cat * > /dev/null
real 0m0.504s
user 0m0.001s
sys 0m0.086s
bash-2.05b# time for x in ls ; do cat $x > /dev/null ; done
real 0m2.415s user 0m0.356s sys 0m1.550s
-- Kenny Dail kend@amigo.net
At 09:55 AM 2/2/2007, Kenny Dail wrote:
What's sad with these tests is it takes 2m15s to read all the data in 71570 individual files, and only 15s to do it in 1 file. Granted there's a bit of overhead from the shell script looping, but not *that* much! I wonder how these tests would go on a different file system. I'm currently using reiserfs.
actually the loop adds a lot of overhead: These commands were run right in a row so there is some speedup due to cacheing, So I repeted test a few times. Admittedly, this is a small maildir.
bash-2.05b# ls | wc 1009 1009 75500
bash-2.05b# time for x in
ls; do cat $x > /dev/null ; donereal 0m3.670s user 0m0.347s sys 0m1.508s bash-2.05b# time cat * > /dev/null
real 0m0.497s user 0m0.015s sys 0m0.071s bash-2.05b# time for x in
ls; do cat $x > /dev/null ; donereal 0m2.237s user 0m0.356s sys 0m1.435s bash-2.05b# time cat * > /dev/null
real 0m0.504s user 0m0.001s sys 0m0.086s bash-2.05b# time for x in
ls; do cat $x > /dev/null ; donereal 0m2.415s user 0m0.356s sys 0m1.550s
wow, that is a ton of extra overhead. I'm doing some testing on ext3 vs reiserfs with different options and it seems that a lot of it has to do with the filesystem and how well it handles tons of files in a directory.
What I've gotten so far using the same shell loop with 71569 small files in the dir. Between each test the mount was unmounted and remounted to clear cache. ext3 defaults: 12m3s (~400k/sec of writes going on) ext3 noatime: 11m31s (0kb/sec writes) ext3 noatime,nodiratime: 11m43s (0k/sec of writes)
reiserfs defaults: 13m35s (~500k/sec of writes) reiserfs noatime: 8m47s (0k/sec of writes) reiserfs noatime,nodiratime: 9m51s (0k/sec of writes) reiserfs noatime,nodiratime,notail: 9m59s (0k/sec writes)
The clear winner seems to be reiserfs with only the noatime option set. I don't know why nodiratime actually slows it down (i ran these tests multiple times to confirm it wasn't flukish results), but perhaps it's an optimization issue with reiserfs.
- Nate
On 1.2.2007, at 3.39, Nate wrote:
- Third connection. It takes ~40 seconds to display a LIST after
login. The index files are untouched; however, during the 40
seconds I'm waiting, I can catch a few of these in lsof. pop3 1971 postfix 10r REG 8,3 4196 205963 /var/spool/ mail/test.visp.net/natetest/cur/1157863512.V902I1f3867.s3.visp.net:2,SThis tells me even though there is no new email, it is re-scanning
the cur/ directory somewhat. It doesn't take near as long as the
initial index build, but certainly takes longer than it would for
the system to read any or all of the listed index files above. Why
would dovecot need to even enter cur/ if the index file is up to
date? Is it verifying that the index file is up to date by re- reading some or all of the many files in cur/? If so, could it not
get the info it needs by simply doing the equivalent of a ls, which
would take considerably less time?
The reason why the messages are read in the first place is because
the message sizes need to be returned so that linefeeds are counted
in CR+LF format, while they're typically stored in LF-only format. So
if Dovecot just stat()ed the file and returned that as the message's
size, it would be violating the POP3 spec. tpop3d seems to be doing
that.
It sounds like dovecot.index.cache file doesn't contain all the
messages' sizes for some reason, and Dovecot goes and re-reads those
files again to get the sizes. I'm not sure why it wouldn't contain
the sizes though, so it sounds like a bug somewhere.
I think this fixes it: http://dovecot.org/list/dovecot-cvs/2007- February/007582.html
Quoting Timo Sirainen:
The reason why the messages are read in the first place is because the message sizes need to be returned so that linefeeds are counted in CR+LF format, while they're typically stored in LF-only format. So if Dovecot just stat()ed the file and returned that as the message's size, it would be violating the POP3 spec. tpop3d seems to be doing that.
Oh. I would suggest a config switch (yes, yet another one) that tells dovecot to do the same. Most clients don't care about it, anyway, and if you tell your LDA to use crlf, dovecot would even do the right thing. It would also enable mail_save_crlf (we could even drop that one). Hm, how does dovecot know that a message contains crlf, so it can use sendfile instead of read/write? If you have some kind of simple autodetection, like "read the first line, if it contains crlf, assume all other lines also do", we already have some (non-optimized) form of that. I guess this also affects IMAP, right?
On 2.2.2007, at 1.29, Jakob Hirsch wrote:
Quoting Timo Sirainen:
The reason why the messages are read in the first place is because
the message sizes need to be returned so that linefeeds are counted in
CR+LF format, while they're typically stored in LF-only format. So if
Dovecot just stat()ed the file and returned that as the message's size, it
would be violating the POP3 spec. tpop3d seems to be doing that.Oh. I would suggest a config switch (yes, yet another one) that tells dovecot to do the same. Most clients don't care about it, anyway,
and if you tell your LDA to use crlf, dovecot would even do the right thing.
Maybe.. I think plugin would be best though, since I don't really
like adding options to intentionally break RFC.
Anyway I don't think it should be that problematic if the cache file
works properly. It probably doesn't even add all that much extra disk
I/O since the client will just download the new mails anyway, so as
long as they stay in kernel's buffer cache long enough there's no
problem. Well, except for a bit of extra CPU usage.
Oh and CVS HEAD deliver stores the message sizes directly into cache,
so it won't be a problem at all with it. :)
Hm, how does dovecot know that a message contains crlf, so it can use sendfile instead of read/write? If you have some kind of simple autodetection, like "read the first line, if it contains crlf, assume all other lines also do", we already have some (non-optimized) form
of that. I guess this also affects IMAP, right?
There's no simple autodetection. There's a "message is binary safe"
flag in cache file that gets set after the message has once been
parsed and only CRLF-linefeeds were seen.
With IMAP the message's size is needed to be sent before the actual
message contents, and it must be exactly correct or the whole session
will break.
Quoting Timo Sirainen:
Oh. I would suggest a config switch (yes, yet another one) that tells dovecot to do the same. Most clients don't care about it, anyway, and if you tell your LDA to use crlf, dovecot would even do the right thing. Maybe.. I think plugin would be best though, since I don't really like adding options to intentionally break RFC.
It would not break RFC if the LDA adds crlf (e.g., Exim's builtin MDA can do that).
Anyway I don't think it should be that problematic if the cache file works properly. It probably doesn't even add all that much extra disk I/O since the client will just download the new mails anyway, so as long as they stay in kernel's buffer cache long enough there's no problem. Well, except for a bit of extra CPU usage.
That's all ok at the average site (and therefore most installations), but it could give some benefit to large sites. But they should be able to pay for such a feature, so we probably don't have to care about that. :)
Oh and CVS HEAD deliver stores the message sizes directly into cache, so it won't be a problem at all with it. :)
Oh. I was already tempted to use dovecot-deliver, now even more :) The only thing I really dislike: Why do deliver send it's own bounces? As I understand a LDA, it tells the MTA by means of the exit code if the delivery was successful or not, so the MTA can decide what to do (e.g. after tempfail, try again later or freeze the message). And I don't even know under which circumstances deliver bounces, as it appears to sometimes use the tempfail exit code.
Hm, how does dovecot know that a message contains crlf, so it can use sendfile instead of read/write? If you have some kind of simple autodetection, like "read the first line, if it contains crlf, assume all other lines also do", we already have some (non-optimized) form of that. I guess this also affects IMAP, right? There's no simple autodetection. There's a "message is binary safe" flag in cache file that gets set after the message has once been parsed and only CRLF-linefeeds were seen.
Oh. So dovecot-deliver could already do that? (maybe it already honours the mail_save_crlf?)
On Fri, 2007-02-02 at 01:03 +0100, Jakob Hirsch wrote:
Hm, how does dovecot know that a message contains crlf, so it can use sendfile instead of read/write? If you have some kind of simple autodetection, like "read the first line, if it contains crlf, assume all other lines also do", we already have some (non-optimized) form of that. I guess this also affects IMAP, right? There's no simple autodetection. There's a "message is binary safe" flag in cache file that gets set after the message has once been parsed and only CRLF-linefeeds were seen.
Oh. So dovecot-deliver could already do that? (maybe it already honours the mail_save_crlf?)
Deliver doesn't update the cache file at all in v1.0 branch. It does use mail_save_crlf though.
On Fri, Feb 02, 2007 at 12:24:42AM +0200, Timo Sirainen wrote:
On 1.2.2007, at 3.39, Nate wrote:
The reason why the messages are read in the first place is because
the message sizes need to be returned so that linefeeds are counted
in CR+LF format, while they're typically stored in LF-only format. So
if Dovecot just stat()ed the file and returned that as the message's
size, it would be violating the POP3 spec. tpop3d seems to be doing
that.
Wouldn't it make sense to add the line count field to the index file? That would solve the CR+LF issue.
And I have another idea, which some time ago saved my butt with Courier: add a line count L=?? maildir flag, which would make messages look like:
-rw------- 1 root root 4811 Feb 9 2006 1138386088.22293_0.mail1,S=4811,L=104
(we had to patch Courier to use that)
This flag could be set by LDA or when generating the index. Wouldn't this solve the issue in an elegant way?
-- Jan Srzednicki w@expro.pl
On Fri, 2007-02-02 at 12:42 +0100, Jan Srzednicki wrote:
On Fri, Feb 02, 2007 at 12:24:42AM +0200, Timo Sirainen wrote:
On 1.2.2007, at 3.39, Nate wrote:
The reason why the messages are read in the first place is because
the message sizes need to be returned so that linefeeds are counted
in CR+LF format, while they're typically stored in LF-only format. So
if Dovecot just stat()ed the file and returned that as the message's
size, it would be violating the POP3 spec. tpop3d seems to be doing
that.Wouldn't it make sense to add the line count field to the index file? That would solve the CR+LF issue.
Cache file already contains the message's virtual size (linefeeds as CRLF), so after once counting the virtual size it shouldn't open the maildir file again unless there's a bug (as in this case there was).
And I have another idea, which some time ago saved my butt with Courier: add a line count L=?? maildir flag, which would make messages look like:
-rw------- 1 root root 4811 Feb 9 2006 1138386088.22293_0.mail1,S=4811,L=104
Dovecot supports reading the virtual size from ,W=1234 in the filename.
On Fri, Feb 02, 2007 at 02:14:47PM +0200, Timo Sirainen wrote:
On Fri, 2007-02-02 at 12:42 +0100, Jan Srzednicki wrote:
Wouldn't it make sense to add the line count field to the index file? That would solve the CR+LF issue.
Cache file already contains the message's virtual size (linefeeds as CRLF), so after once counting the virtual size it shouldn't open the maildir file again unless there's a bug (as in this case there was).
ACK, I see.
And I have another idea, which some time ago saved my butt with Courier: add a line count L=?? maildir flag, which would make messages look like:
-rw------- 1 root root 4811 Feb 9 2006 1138386088.22293_0.mail1,S=4811,L=104
Dovecot supports reading the virtual size from ,W=1234 in the filename.
It doesn't seem to use/set it by default, how can I make it to?
-- Jan Srzednicki w@expro.pl
On Fri, 2007-02-02 at 13:17 +0100, Jan Srzednicki wrote:
And I have another idea, which some time ago saved my butt with Courier: add a line count L=?? maildir flag, which would make messages look like:
-rw------- 1 root root 4811 Feb 9 2006 1138386088.22293_0.mail1,S=4811,L=104
Dovecot supports reading the virtual size from ,W=1234 in the filename.
It doesn't seem to use/set it by default, how can I make it to?
1.0-branch Dovecot LDA doesn't do that, since it doesn't really support parsing the message while saving it.
1.1-branch Dovecot LDA does support parsing message while saving it, but in that case the ,W= isn't really needed because the size is saved into cache file anyway.
I think at least Exim supports adding the ,W= to the maildir file.
On Fri, Feb 02, 2007 at 02:40:34PM +0200, Timo Sirainen wrote:
On Fri, 2007-02-02 at 13:17 +0100, Jan Srzednicki wrote:
1.0-branch Dovecot LDA doesn't do that, since it doesn't really support parsing the message while saving it.
Would there be an easy patch for dovecot LDA to support that?
And, the flags could be also set during reindexing by the POP3/IMAP daemon.
-- Jan Srzednicki w@expro.pl
On Fri, 2007-02-02 at 13:49 +0100, Jan Srzednicki wrote:
On Fri, Feb 02, 2007 at 02:40:34PM +0200, Timo Sirainen wrote:
On Fri, 2007-02-02 at 13:17 +0100, Jan Srzednicki wrote:
1.0-branch Dovecot LDA doesn't do that, since it doesn't really support parsing the message while saving it.
Would there be an easy patch for dovecot LDA to support that?
It might not be that difficult to kludge if you really wanted to, but I don't really want to do that.
And, the flags could be also set during reindexing by the POP3/IMAP daemon.
That would change the maildir base filename. Not a very good idea to do.
On Fri, 2007-02-02 at 14:40 +0200, Timo Sirainen wrote:
1.1-branch Dovecot LDA does support parsing message while saving it, but in that case the ,W= isn't really needed because the size is saved into cache file anyway.
Would it be useful to save it in the filename to avoid reading all of the file data if the cache had to be rebuilt, or would you need to read the file anyway?
Richard
On Fri, 2007-02-02 at 17:04 -0600, Richard Laager wrote:
On Fri, 2007-02-02 at 14:40 +0200, Timo Sirainen wrote:
1.1-branch Dovecot LDA does support parsing message while saving it, but in that case the ,W= isn't really needed because the size is saved into cache file anyway.
Would it be useful to save it in the filename to avoid reading all of the file data if the cache had to be rebuilt, or would you need to read the file anyway?
In normal Dovecot operation the cache never needs to be rebuilt from scratch.. But if in your environment the cache file for some reason gets lost, then W= could benefit, especially with POP3. With IMAP it would depend on the client.
Quoting Timo Sirainen:
(oops, forgot list cc...)
Dovecot supports reading the virtual size from ,W=1234 in the filename.
Oh. Is this documented somewhere? And are there any other filename tags?
I think at least Exim supports adding the ,W= to the maildir file.
Should be something like this in the appendfile transport for dovecot: maildir_tag = W=${eval:$message_size+$message_linecount}
to get filenames like new/1170546076.H147536P16502.ymmv.de:W=354 (Adding $message_linecount with eval: is only needed when the file contains only LFs, of course.)
Or should it be "...:,W=354"?
What happens if the number is wrong? If dovecot trusts in it, it would break the imap session, I guess, but would dovecot complain about it?
On Sun, 2007-02-04 at 00:56 +0100, Jakob Hirsch wrote:
Quoting Timo Sirainen:
(oops, forgot list cc...)
Dovecot supports reading the virtual size from ,W=1234 in the filename.
Oh. Is this documented somewhere? And are there any other filename tags?
Not documented. I think some other software also used this. 'S' and 'W' are the only flags that Dovecot supports.
I think at least Exim supports adding the ,W= to the maildir file.
Should be something like this in the appendfile transport for dovecot: maildir_tag = W=${eval:$message_size+$message_linecount}
to get filenames like new/1170546076.H147536P16502.ymmv.de:W=354 (Adding $message_linecount with eval: is only needed when the file contains only LFs, of course.)
Or should it be "...:,W=354"?
No, there must not be ':'. So ...,W=1234 is correct.
What happens if the number is wrong? If dovecot trusts in it, it would break the imap session, I guess, but would dovecot complain about it?
Before rc20 Dovecot disconnects the client when fetching the message, but I don't think it actually logs any error message. With rc20 and later it logs an error:
FETCH for mailbox %s UID %u got too little data: %u vs %u
Quoting Timo Sirainen:
maildir_tag = W=${eval:$message_size+$message_linecount}
This does not work, as $message_linecount is not correct (obviously, some headers are not counted). Probably a bug, I'll have a look there.
to get filenames like new/1170546076.H147536P16502.ymmv.de:W=354 Or should it be "...:,W=354"? No, there must not be ':'. So ...,W=1234 is correct.
Oh. Unfortunately, Exim always adds ":" after the the "1170767045.H941467P3018.ymmv.de" part when maildir_tag is set.
Shouldn't it be "...ymmv.de:2,W=1234" to conform with the Maildir++ spec?
On Tue, 2007-02-06 at 14:11 +0100, Jakob Hirsch wrote:
Shouldn't it be "...ymmv.de:2,W=1234" to conform with the Maildir++ spec?
No, everything after :2, are flags. So that would mean that there are flags 'W', '=', '1', '2', '3' and '4'.
Dovecot however supports some kind of "extra flag fields" even though they're not maildir standard and other maildir clients could mess them up. You could have:
ymmv.de:2,,W=1234
Normal message flags would go between the commas. Or at least I think that would work, don't remember the last time I tested it.
At 02:24 PM 2/1/2007, Timo Sirainen wrote:
The reason why the messages are read in the first place is because the message sizes need to be returned so that linefeeds are counted in CR+LF format, while they're typically stored in LF-only format. So if Dovecot just stat()ed the file and returned that as the message's size, it would be violating the POP3 spec. tpop3d seems to be doing that.
It sounds like dovecot.index.cache file doesn't contain all the messages' sizes for some reason, and Dovecot goes and re-reads those files again to get the sizes. I'm not sure why it wouldn't contain the sizes though, so it sounds like a bug somewhere.
I think this fixes it: http://dovecot.org/list/dovecot-cvs/2007- February/007582.html
When I tried the 2 indexing patches against the 1/31/07 snapshot I had the same problems as reported with the rc20 release, so I installed the rc21 release and confirmed the re-reading of the cur/ directory was eliminated! VERY cool!
Now installed on production pop3 server, and large mailboxes (10000+ messages) load in fractions of seconds now rather than taking minutes! I/O load on the server is also shrinking back to normal. Previous load averages hovered around 6.00. After updating from rc19 to rc21 load averages dropped and remain below 1.00, and with significantly better performance than tpop3d.
I would say for medium to large capacity sites this is a major performance increase and should be listed as such in Changelog.
Thank you!
- Nate
participants (8)
-
FiL
-
FiL ＠ Kpoxa
-
Jakob Hirsch
-
Jan Srzednicki
-
Kenny Dail
-
Nate
-
Richard Laager
-
Timo Sirainen