We recently upgraded from Dovecot 1.2.15 (on Debian squeeze) to Dovecot 2.1.7 (on Debian wheezy/stable).

For now, we continue to use FTS with Squat indexes. We'd like to eventually switch to Lucene but it's not yet supported in Debian stable.

Since upgrading to 2.1.7, we've seen some problems with IMAP search results when an FTS Squat index is present (i.e., when the "dovecot.index.search" and "dovecot.index.search.uids" files are present). If it's present, even header searches sometimes give incorrect results (such as searches on FROM or SUBJECT headers).

I'm wondering if this is a known issue with 2.1.7 and whether upgrading to a later version of Dovecot with Squat is likely to fix this (due to, say, http://www.dovecot.org/list/dovecot-cvs/2012-June/020881.html), or if Squat in all versions is a lost cause and I need to switch to Lucene.

I can demonstrate the problem by showing the results of three searches on my mailbox. First, here are the correct results for header FTS searches when no "dovecot.index.search[.uids]" file is present:

. UID SEARCH FROM "Nick"

• SEARCH 9 66 100 116 179 212 274 338 347 386 494 519 579 603 643 766 781 788 827 1107 1280 1626 1641 1667 1688 1702 1703 2184 2187 3202 3519 3716 3798 3885 3886 4024 4166 4345 4400 4401 4436 4491 4561 4595 4618 4676 4711 4712 4746 4788 4883 5217 5253 5577 5602 5722 . OK Search completed (0.059 secs).

. UID SEARCH ALL FROM "Nick"

• SEARCH 9 66 100 116 179 212 274 338 347 386 494 519 579 603 643 766 781 788 827 1107 1280 1626 1641 1667 1688 1702 1703 2184 2187 3202 3519 3716 3798 3885 3886 4024 4166 4345 4400 4401 4436 4491 4561 4595 4618 4676 4711 4712 4746 4788 4883 5217 5253 5577 5602 5722 . OK Search completed (0.058 secs).

. UID SEARCH SUBJECT "Nick"

• SEARCH 3466 3467 3507 . OK Search completed (0.054 secs).

These results are consistent (and correct) whether "mail_plugins = fts fts_squat" is enabled or not, as long as no "dovecot.index.search[.uids]" file is present.

Now I generate an FTS index:

. UID SEARCH TEXT "Nick"

• OK Indexed 98% of the mailbox, ETA 0:00
• OK Mailbox indexing finished
• SEARCH 9 32 66 100 111 112 113 116 179 199 212 228 229 240 274 303 338 347 386 393 448 478 491 494 519 579 591 603 614 618 624 643 649 742 766 781 788 827 837 839 841 1095 1106 1107 1174 1188 1225 1227 1228 1240 1254 1261 1278 1280 1452 1482 1519 1520 1595 1626 1641 1656 1667 1688 1702 1703 1724 1758 2184 2187 2798 2883 2932 3034 3198 3202 3206 3230 3371 3432 3454 3457 3466 3467 3507 3519 3631 3637 3685 3702 3703 3716 3765 3798 3799 3803 3853 3854 3855 3857 3873 3885 3886 3962 3969 4024 4033 4050 4051 4058 4161 4166 4173 4176 4224 4273 4345 4400 4401 4436 4461 4490 4491 4498 4561 4595 4618 4651 4676 4711 4712 4713 4729 4731 4736 4737 4738 4739 4746 4786 4787 4788 4813 4816 4822 4847 4883 4885 4933 4941 4942 4957 4958 5029 5052 5097 5135 5136 5182 5217 5253 5271 5357 5543 5551 5565 5577 5589 5602 5612 5643 5692 5722 . OK Search completed (11.078 secs).

Now I run the same original three searches, but they return completely different results(!):

. UID SEARCH FROM "Nick"

• SEARCH 9 66 100 116 179 212 228 240 274 338 347 386 448 494 519 579 603 643 766 781 788 827 1095 1107 1174 1240 1254 1261 1280 1595 1626 1641 1667 1688 1702 1703 1724 2184 2187 2883 2932 3202 3371 3432 3457 3466 3467 3507 3519 3631 3637 3685 3716 3765 3798 3803 3853 3854 3855 3885 3886 3969 4024 4050 4166 4176 4224 4345 4400 4401 4436 4461 4490 4491 4498 4561 4595 4618 4676 4711 4712 4729 4731 4746 4788 4883 5029 5135 5136 5182 5217 5253 5577 5602 5722 . OK Search completed (0.001 secs).

. UID SEARCH ALL FROM "Nick"

• SEARCH . OK Search completed (0.001 secs).

. UID SEARCH SUBJECT "Nick"

• SEARCH 9 66 100 116 179 212 228 240 274 338 347 386 448 494 519 579 603 643 766 781 788 827 1095 1107 1174 1240 1254 1261 1280 1595 1626 1641 1667 1688 1702 1703 1724 2184 2187 2883 2932 3202 3371 3432 3457 3466 3467 3507 3519 3631 3637 3685 3716 3765 3798 3803 3853 3854 3855 3885 3886 3969 4024 4050 4166 4176 4224 4345 4400 4401 4436 4461 4490 4491 4498 4561 4595 4618 4676 4711 4712 4729 4731 4746 4788 4883 5029 5135 5136 5182 5217 5253 5577 5602 5722 . OK Search completed (0.001 secs).

The "ALL" one returns nothing at all, even though an "AND" search on "ALL" should make no difference, and the "FROM" and "SUBJECT" ones now return identical incorrect results, many of which do not actually match the header requested when examined -- it's returning messages with "Nick" in *any* header. For example, message 9 does not contain "Nick" in the subject, despite the results shown above:

. FETCH 9 UID

• 9 FETCH (UID 9) . OK Fetch completed.

FETCH 9 BODY.PEEK[HEADER.FIELDS (SUBJECT)]

• 9 FETCH (BODY[HEADER.FIELDS (SUBJECT)] {47} Subject: Re: Finishing Gallery support page

) . OK Fetch completed.

If I delete the "dovecot.index.search" and "dovecot.index.search.uids" files, or if I disable fts in the configuration entirely, the results immediately become correct again.

As I mentioned above, I'm wondering if anyone knows the cause of (and fix for) this.

Here's my doveconf -n:

# 2.1.7: /etc/dovecot/dovecot.conf # OS: Linux 3.2.0-4-amd64 x86_64 Debian 7.4 auth_master_user_separator = * auth_mechanisms = plain login auth_username_format = default_vsz_limit = 1 G disable_plaintext_auth = no lda_mailbox_autosubscribe = yes listen = * log_timestamp = "%Y-%m-%d %H:%M:%S " login_log_format_elements = pid=[%p] user=<%u> method=%m rip=%r lip=%l %c mail_fsync = never mail_location = maildir:~/ mail_max_userip_connections = 100 mail_plugins = fts fts_squat mail_log notify managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave spamtest spamtestplus namespace { inbox = yes location = prefix = INBOX. separator = . type = private } namespace { hidden = yes inbox = no list = no location = prefix = separator = . type = private } passdb { args = /etc/dovecot/dovecot.users driver = passwd-file } passdb { args = /etc/dovecot/master-passwords driver = passwd-file master = yes pass = yes } plugin { fts = squat mail_log_events = copy expunge mail_log_fields = box msgid size from sieve = %h/mailbox.sieve sieve_after = %h/../../domain-after.sieve sieve_before = %h/../../domain-before.sieve sieve_dir = %h/sieve sieve_extensions = -reject +spamtest +spamtestplus sieve_global_dir = %h/../../sieve-global-include-scripts sieve_spamtest_max_value = 7 sieve_spamtest_status_header = X-Spam-Level sieve_spamtest_status_type = strlen } protocols = pop3 imap lmtp sieve service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0660 user = postfix } } service imap-login { process_min_avail = 8 service_count = 0 } service imap { process_limit = 2000 } service lmtp { unix_listener /var/spool/postfix/private/dovecot-lmtp { group = postfix mode = 0660 user = postfix } } service pop3-login { process_min_avail = 8 service_count = 0 } service pop3 { process_limit = 2000 } ssl = no userdb { args = /etc/dovecot/dovecot.users driver = passwd-file } verbose_proctitle = yes protocol imap { imap_client_workarounds = delay-newmail imap_logout_format = %i bytes in, %o bytes out } protocol pop3 { pop3_client_workarounds = outlook-no-nuls oe-ns-eoh pop3_logout_format = ; %i bytes in, %o bytes out, %t top (%p bytes), %r retr (%b bytes), %d of %m deleted, mailbox size %s bytes pop3_uidl_format = UID%u-%v } protocol lda { mail_fsync = optimized mail_plugins = sieve postmaster_address = postmaster@tigertech.net sendmail_path = /usr/local/bin/dovecot-sendmail-wrapper } protocol lmtp { mail_fsync = optimized mail_plugins = sieve postmaster_address = postmaster@tigertech.net sendmail_path = /usr/local/bin/dovecot-sendmail-wrapper }

-- Robert L Mathews, Tiger Technologies, http://www.tigertech.net/