[Dovecot] body search very slow since upgrade from 1.0.15 to 1.2.10
Did you mislead me Timo? You said search in 1.1+ is faster than 1.0. I'm seeing approximately 20x *slower* search times in 1.2.10.
Via Thunderbird, a full body search of my 11,000+ message IMAP folder hosted by 1.0.15 used to take less than 10 seconds. Since upgrading to 1.2.10 the search is taking over 3 minutes, the imap process servicing the client pegging one CPU at 100% for the duration of the search. Here's the mbox file:
-rw------- 1 stan stan 50M Feb 23 07:41 1-Spam-l
Doing the same search via grep results in a time of:
real 0m0.361s user 0m0.204s sys 0m0.140s
grep takes less than half a second, yet dovecot 1.2.10 takes 3 minutes for the same search? Actually grep is searching the entire file including headers, so it's doing more work than the imap process. Something is *SERIOUSLY* wrong here.
Did I somehow nerfed something in dovecot.conf that's causing this? I deleted all the index files as per the upgrade instructions. However, checking the directories shows they're apparently rebuilt:
[08:09:15][root@greer]/home/stan/mail/.imap/1-Spam-l$ la total 28M drwx------ 2 stan stan 4.0K Feb 23 07:41 . drwx------ 35 stan stan 4.0K Feb 15 11:14 .. -rw------- 1 stan stan 358K Feb 23 07:18 dovecot.index -rw------- 1 stan stan 27M Feb 23 07:41 dovecot.index.cache -rw------- 1 stan stan 616 Feb 23 07:41 dovecot.index.log
What do I need to check/change/do to get these search times back down to reasonable/respectable levels, i.e. around 10s?
Thanks.
-- Stan
On 23.2.2010, at 16.19, Stan Hoeppner wrote:
Did you mislead me Timo? You said search in 1.1+ is faster than 1.0. I'm seeing approximately 20x *slower* search times in 1.2.10.
Via Thunderbird, a full body search of my 11,000+ message IMAP folder hosted by 1.0.15 used to take less than 10 seconds. Since upgrading to 1.2.10 the search is taking over 3 minutes, the imap process servicing the client pegging one CPU at 100% for the duration of the search. Here's the mbox file:
Well, v1.1+ does case-insensitive searches by converting all mails to UTF-8 first. But it really shouldn't make it that much slower. What OS and CPU is this with?
You could also try if it happens to work faster by compiling Dovecot without iconv support.
I guess there could also be some kind of a bug that you're triggering.
Timo Sirainen put forth on 2/23/2010 12:33 PM:
On 23.2.2010, at 16.19, Stan Hoeppner wrote:
Did you mislead me Timo? You said search in 1.1+ is faster than 1.0. I'm seeing approximately 20x *slower* search times in 1.2.10.
Via Thunderbird, a full body search of my 11,000+ message IMAP folder hosted by 1.0.15 used to take less than 10 seconds. Since upgrading to 1.2.10 the search is taking over 3 minutes, the imap process servicing the client pegging one CPU at 100% for the duration of the search. Here's the mbox file:
Well, v1.1+ does case-insensitive searches by converting all mails to UTF-8 first. But it really shouldn't make it that much slower. What OS and CPU is this with?
Debian 5.0.4 Linux kernel 2.6.31.1 rolled by me from kernel.org source Dovecot, Debian backport Version: 1:1.2.10-1~bpo50+1 Intel 440BX Abit BP6 mobo, Dual Mendocino Celerons @550 MHz, 384MB PC100
You could also try if it happens to work faster by compiling Dovecot without iconv support.
Was iconv support in dovecot 1.0.15? How can I check to see if my dovecot 1.2.10 already has iconv support? Is there a dovecot command similar to postconf -m which shows what database types were compiled into the postfix binaries?
I guess there could also be some kind of a bug that you're triggering.
Is there any setting in dovecot.conf, that if set incorrectly, could cause this? If not, and assuming the Debian team didn't screw up the package, I'd lean toward this bug explanation, except that no one else is reporting this problem, are they?
These searches used to take about 10 seconds on 1.0.15. They're taking 3 minutes or more on 1.2.10. Also, on 1.0.15, if I ran the same search back-to-back, the second search completed _instantly_. On 1.2.10, the second search takes as long as the first. This leads me to believe the indexes and caches aren't being used at all during the searches, or something along these lines. Shouldn't they be?
Again, if grep can do these searches in less than a second, there's no reason why imap can't do them in 10 seconds. Something has got to be horribly wrong here...
Tell me what I need to do to fix this, whilst not having to go outside the Debian package system.
Thanks.
-- Stan
On 23.2.2010, at 22.49, Stan Hoeppner wrote:
Well, v1.1+ does case-insensitive searches by converting all mails to UTF-8 first. But it really shouldn't make it that much slower. What OS and CPU is this with?
Debian 5.0.4 Linux kernel 2.6.31.1 rolled by me from kernel.org source Dovecot, Debian backport Version: 1:1.2.10-1~bpo50+1 Intel 440BX Abit BP6 mobo, Dual Mendocino Celerons @550 MHz, 384MB PC100
So not really the fastest CPU :)
You could also try if it happens to work faster by compiling Dovecot without iconv support.
Was iconv support in dovecot 1.0.15? How can I check to see if my dovecot 1.2.10 already has iconv support? Is there a dovecot command similar to postconf -m which shows what database types were compiled into the postfix binaries?
Actually now that I think of it, iconv was used in v1.0 just the same. The difference is just the extra step of converting all text to titlecase before comparing it. That should be a pretty fast operation too.
I guess there could also be some kind of a bug that you're triggering.
Is there any setting in dovecot.conf, that if set incorrectly, could cause this?
No.
If not, and assuming the Debian team didn't screw up the package, I'd lean toward this bug explanation, except that no one else is reporting this problem, are they?
I haven't heard. But then again most people don't seem to be using server-side SEARCH anyway, especially on older machines.
These searches used to take about 10 seconds on 1.0.15. They're taking 3 minutes or more on 1.2.10. Also, on 1.0.15, if I ran the same search back-to-back, the second search completed _instantly_. On 1.2.10, the second search takes as long as the first. This leads me to believe the indexes and caches aren't being used at all during the searches, or something along these lines. Shouldn't they be?
No, body searches aren't indexed (without fts plugins).
This happened for some spam mailbox? Could you send it to me and I'll try if I can reproduce it?
participants (2)
-
Stan Hoeppner
-
Timo Sirainen