[Dovecot] Search for (any of) multiple terms slow

[Patrick Nagel]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi,

this may be an obvious logical problem I'm not aware of, which cannot be solved
any more efficiently... but maybe it's just a bug or there is potential for
optimisation in Dovecot (or Thunderbird?).

When searching for multiple terms at once ("any of") with Thunderbird/Dovecot
(using FTS Squat indexes), it takes much longer (not double the time, or a bit
more than that, but really much, much longer) than when I search for only one term.

Here are some figures (the IMAP commands are shown as issued by Thunderbird and
recorded by Dovecot rawlog).

While searching, I can see the 'imap' process using 100% CPU time.

Searching for "xyz" in Archive and subfolders
4 secs

32 select "Archive"
33 uid SEARCH UNDELETED BODY "xyz"
34 select "Archive.2004"
35 UID fetch 1:* (FLAGS)
36 uid SEARCH UNDELETED BODY "xyz"
37 select "Archive.2004.2005"
38 UID fetch 1:* (FLAGS)
39 uid SEARCH UNDELETED BODY "xyz"
40 select "Archive.2004.2005.2006"
41 UID fetch 1:* (FLAGS)
42 uid SEARCH UNDELETED BODY "xyz"
43 select "Archive.2004.2005.2006.2007"
44 UID fetch 1:* (FLAGS)
45 uid SEARCH UNDELETED BODY "xyz"
46 select "Archive.2004.2005.2006.2007.2008"
47 UID fetch 1:* (FLAGS)
48 uid SEARCH UNDELETED BODY "xyz"
49 IDLE

Searching for "xyz" and "abc" (any of) in Archive and subfolders
123 secs

50 select "Archive"
51 uid SEARCH UNDELETED (OR BODY "xyz" BODY "abc")
52 select "Archive.2004"
53 UID fetch 1:* (FLAGS)
54 uid SEARCH UNDELETED (OR BODY "xyz" BODY "abc")
55 select "Archive.2004.2005"
56 UID fetch 1:* (FLAGS)
57 uid SEARCH UNDELETED (OR BODY "xyz" BODY "abc")
58 select "Archive.2004.2005.2006"
59 UID fetch 1:* (FLAGS)
60 uid SEARCH UNDELETED (OR BODY "xyz" BODY "abc")
61 select "Archive.2004.2005.2006.2007"
62 UID fetch 1:* (FLAGS)
63 uid SEARCH UNDELETED (OR BODY "xyz" BODY "abc")
64 select "Archive.2004.2005.2006.2007.2008"
65 UID fetch 1:* (FLAGS)
66 uid SEARCH UNDELETED (OR BODY "xyz" BODY "abc")
67 IDLE

Searching for "xyz", "abc" and "def" in Archive and subfolders
173 secs

68 select "Archive"
69 uid SEARCH UNDELETED (OR (OR BODY "xyz" BODY "abc") BODY "def")
70 select "Archive.2004"
71 UID fetch 1:* (FLAGS)
72 uid SEARCH UNDELETED (OR (OR BODY "xyz" BODY "abc") BODY "def")
73 select "Archive.2004.2005"
74 UID fetch 1:* (FLAGS)
75 uid SEARCH UNDELETED (OR (OR BODY "xyz" BODY "abc") BODY "def")
76 select "Archive.2004.2005.2006"
77 UID fetch 1:* (FLAGS)
78 uid SEARCH UNDELETED (OR (OR BODY "xyz" BODY "abc") BODY "def")
79 select "INBOX"
80 UID fetch 1:* (FLAGS)
81 check
82 select "Archive.2004.2005.2006.2007"
83 UID fetch 1:* (FLAGS)
84 uid SEARCH UNDELETED (OR (OR BODY "xyz" BODY "abc") BODY "def")
85 select "Archive.2004.2005.2006.2007.2008"
86 UID fetch 1:* (FLAGS)
87 uid SEARCH UNDELETED (OR (OR BODY "xyz" BODY "abc") BODY "def")
88 IDLE

Searching for "xyz", "abc", "def" and "ghi" in Archive and subfolders
217 secs

89 select "Archive"
90 uid SEARCH UNDELETED (OR (OR (OR BODY "xyz" BODY "abc") BODY "def") BODY "ghi")
91 select "Archive.2004"
92 UID fetch 1:* (FLAGS)
93 uid SEARCH UNDELETED (OR (OR (OR BODY "xyz" BODY "abc") BODY "def") BODY "ghi")
94 select "Archive.2004.2005"
95 UID fetch 1:* (FLAGS)
96 uid SEARCH UNDELETED (OR (OR (OR BODY "xyz" BODY "abc") BODY "def") BODY "ghi")
97 select "Archive.2004.2005.2006"
98 UID fetch 1:* (FLAGS)
99 uid SEARCH UNDELETED (OR (OR (OR BODY "xyz" BODY "abc") BODY "def") BODY "ghi")
100 select "Archive.2004.2005.2006.2007"
101 UID fetch 1:* (FLAGS)
102 uid SEARCH UNDELETED (OR (OR (OR BODY "xyz" BODY "abc") BODY "def") BODY "ghi")
103 select "Archive.2004.2005.2006.2007.2008"
104 UID fetch 1:* (FLAGS)
105 uid SEARCH UNDELETED (OR (OR (OR BODY "xyz" BODY "abc") BODY "def") BODY "ghi")
106 IDLE

Patrick.

- --
STAR Software (Shanghai) Co., Ltd.            http://www.star-group.net/
Phone:    +86 (21) 3462 7688 x 826             Fax:   +86 (21) 3462 7779

PGP key:         https://stshacom1.star-china.net/keys/patrick_nagel.asc
Fingerprint:           E09A D65E 855F B334 E5C3 5386 EF23 20FC E883 A005
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org

iD8DBQFIooxq7yMg/OiDoAURAhb9AJ4jOaK+eHXcSwJTAER8xEKIlJJt0gCgtZan
wJm05V0F03gj+6YxbNOtCew=
=jID/
-----END PGP SIGNATURE-----