[Dovecot] Search for (any of) multiple terms slow
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Hi,
this may be an obvious logical problem I'm not aware of, which cannot be solved any more efficiently... but maybe it's just a bug or there is potential for optimisation in Dovecot (or Thunderbird?).
When searching for multiple terms at once ("any of") with Thunderbird/Dovecot (using FTS Squat indexes), it takes much longer (not double the time, or a bit more than that, but really much, much longer) than when I search for only one term.
Here are some figures (the IMAP commands are shown as issued by Thunderbird and recorded by Dovecot rawlog).
While searching, I can see the 'imap' process using 100% CPU time.
Searching for "xyz" in Archive and subfolders 4 secs
32 select "Archive" 33 uid SEARCH UNDELETED BODY "xyz" 34 select "Archive.2004" 35 UID fetch 1:* (FLAGS) 36 uid SEARCH UNDELETED BODY "xyz" 37 select "Archive.2004.2005" 38 UID fetch 1:* (FLAGS) 39 uid SEARCH UNDELETED BODY "xyz" 40 select "Archive.2004.2005.2006" 41 UID fetch 1:* (FLAGS) 42 uid SEARCH UNDELETED BODY "xyz" 43 select "Archive.2004.2005.2006.2007" 44 UID fetch 1:* (FLAGS) 45 uid SEARCH UNDELETED BODY "xyz" 46 select "Archive.2004.2005.2006.2007.2008" 47 UID fetch 1:* (FLAGS) 48 uid SEARCH UNDELETED BODY "xyz" 49 IDLE
Searching for "xyz" and "abc" (any of) in Archive and subfolders 123 secs
50 select "Archive" 51 uid SEARCH UNDELETED (OR BODY "xyz" BODY "abc") 52 select "Archive.2004" 53 UID fetch 1:* (FLAGS) 54 uid SEARCH UNDELETED (OR BODY "xyz" BODY "abc") 55 select "Archive.2004.2005" 56 UID fetch 1:* (FLAGS) 57 uid SEARCH UNDELETED (OR BODY "xyz" BODY "abc") 58 select "Archive.2004.2005.2006" 59 UID fetch 1:* (FLAGS) 60 uid SEARCH UNDELETED (OR BODY "xyz" BODY "abc") 61 select "Archive.2004.2005.2006.2007" 62 UID fetch 1:* (FLAGS) 63 uid SEARCH UNDELETED (OR BODY "xyz" BODY "abc") 64 select "Archive.2004.2005.2006.2007.2008" 65 UID fetch 1:* (FLAGS) 66 uid SEARCH UNDELETED (OR BODY "xyz" BODY "abc") 67 IDLE
Searching for "xyz", "abc" and "def" in Archive and subfolders 173 secs
68 select "Archive" 69 uid SEARCH UNDELETED (OR (OR BODY "xyz" BODY "abc") BODY "def") 70 select "Archive.2004" 71 UID fetch 1:* (FLAGS) 72 uid SEARCH UNDELETED (OR (OR BODY "xyz" BODY "abc") BODY "def") 73 select "Archive.2004.2005" 74 UID fetch 1:* (FLAGS) 75 uid SEARCH UNDELETED (OR (OR BODY "xyz" BODY "abc") BODY "def") 76 select "Archive.2004.2005.2006" 77 UID fetch 1:* (FLAGS) 78 uid SEARCH UNDELETED (OR (OR BODY "xyz" BODY "abc") BODY "def") 79 select "INBOX" 80 UID fetch 1:* (FLAGS) 81 check 82 select "Archive.2004.2005.2006.2007" 83 UID fetch 1:* (FLAGS) 84 uid SEARCH UNDELETED (OR (OR BODY "xyz" BODY "abc") BODY "def") 85 select "Archive.2004.2005.2006.2007.2008" 86 UID fetch 1:* (FLAGS) 87 uid SEARCH UNDELETED (OR (OR BODY "xyz" BODY "abc") BODY "def") 88 IDLE
Searching for "xyz", "abc", "def" and "ghi" in Archive and subfolders 217 secs
89 select "Archive" 90 uid SEARCH UNDELETED (OR (OR (OR BODY "xyz" BODY "abc") BODY "def") BODY "ghi") 91 select "Archive.2004" 92 UID fetch 1:* (FLAGS) 93 uid SEARCH UNDELETED (OR (OR (OR BODY "xyz" BODY "abc") BODY "def") BODY "ghi") 94 select "Archive.2004.2005" 95 UID fetch 1:* (FLAGS) 96 uid SEARCH UNDELETED (OR (OR (OR BODY "xyz" BODY "abc") BODY "def") BODY "ghi") 97 select "Archive.2004.2005.2006" 98 UID fetch 1:* (FLAGS) 99 uid SEARCH UNDELETED (OR (OR (OR BODY "xyz" BODY "abc") BODY "def") BODY "ghi") 100 select "Archive.2004.2005.2006.2007" 101 UID fetch 1:* (FLAGS) 102 uid SEARCH UNDELETED (OR (OR (OR BODY "xyz" BODY "abc") BODY "def") BODY "ghi") 103 select "Archive.2004.2005.2006.2007.2008" 104 UID fetch 1:* (FLAGS) 105 uid SEARCH UNDELETED (OR (OR (OR BODY "xyz" BODY "abc") BODY "def") BODY "ghi") 106 IDLE
Patrick.
STAR Software (Shanghai) Co., Ltd. http://www.star-group.net/ Phone: +86 (21) 3462 7688 x 826 Fax: +86 (21) 3462 7779
PGP key: https://stshacom1.star-china.net/keys/patrick_nagel.asc Fingerprint: E09A D65E 855F B334 E5C3 5386 EF23 20FC E883 A005 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org
iD8DBQFIooxq7yMg/OiDoAURAhb9AJ4jOaK+eHXcSwJTAER8xEKIlJJt0gCgtZan wJm05V0F03gj+6YxbNOtCew= =jID/ -----END PGP SIGNATURE-----
On Aug 13, 2008, at 3:25 AM, Patrick Nagel wrote:
this may be an obvious logical problem I'm not aware of, which
cannot be solved any more efficiently... but maybe it's just a bug or there is
potential for optimisation in Dovecot (or Thunderbird?).When searching for multiple terms at once ("any of") with
Thunderbird/Dovecot (using FTS Squat indexes), it takes much longer (not double the
time, or a bit more than that, but really much, much longer) than when I search for
only one term.
Yea, OR isn't supported by FTS plugin yet.. I had hoped no-one would
notice it ;) I guess I should do something about it. Requires a lot
more complexity to the code though..
Timo Sirainen wrote:
Yea, ...
I've been meaning to tell you that should be "Yeah" for an informal version of "Yes", otherwise it's a very archaic form of "Yes" or "Indeed" as in "Yea, though I walk in the valley of the shadow of death"!
I think that's also true of American/Aussie etc. as well ...
Best Wishes, Chris
-- --+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+- Christopher Wakelin, c.d.wakelin@reading.ac.uk IT Services Centre, The University of Reading, Tel: +44 (0)118 378 8439 Whiteknights, Reading, RG6 2AF, UK Fax: +44 (0)118 975 3094
On Aug 13, 2008, at 5:24 PM, Chris Wakelin wrote:
Timo Sirainen wrote:
Yea, ...
I've been meaning to tell you that should be "Yeah" for an informal
version of "Yes", otherwise it's a very archaic form of "Yes" or
"Indeed" as in "Yea, though I walk in the valley of the shadow of
death"!
Hmm. I've never paid attention to that. Grepping my IRC logs I seem to
have used that since the beginning. But I can also see a lot of other
people are saying "yea" (but no idea if they're native english
speakers). Wikipedia says it's a common misspelling. Perhaps I should
try to change it. :)
But Timo walks through the valley of the shadow for us all.....so maybe he's entitled.....
Timo Sirainen wrote:
On Aug 13, 2008, at 5:24 PM, Chris Wakelin wrote:
Timo Sirainen wrote:
Yea, ...
I've been meaning to tell you that should be "Yeah" for an informal version of "Yes", otherwise it's a very archaic form of "Yes" or "Indeed" as in "Yea, though I walk in the valley of the shadow of death"!
Hmm. I've never paid attention to that. Grepping my IRC logs I seem to have used that since the beginning. But I can also see a lot of other people are saying "yea" (but no idea if they're native english speakers). Wikipedia says it's a common misspelling. Perhaps I should try to change it. :)
--
Stewart Dean, Unix System Admin, Henderson Computer Resources Center of Bard College, Annandale-on-Hudson, New York 12504 sdean@bard.edu voice: 845-758-7475, fax: 845-758-7035
Chris Wakelin wrote:
Timo Sirainen wrote:
Yea, ...
I've been meaning to tell you that should be "Yeah" for an informal version of "Yes", otherwise it's a very archaic form of "Yes" or "Indeed" as in "Yea, though I walk in the valley of the shadow of death"!
As a great man said, "In fact, now that I think of it, I shall RUN through the valley of the shadow of death, because you get out of the valley quicker that way."
I think that's also true of American/Aussie etc. as well ...
Yep!
Best Wishes, Chris
participants (5)
-
Chris Wakelin
-
Jay Levitt
-
Patrick Nagel
-
Stewart Dean
-
Timo Sirainen