FTS search with CLucene giving odd results/exposing config issue

Wed Mar 11 17:12:29 UTC 2015

I recently got FTS search with Clucene working on FreeBSD 10.1 (I had some trouble with 9.3, postponed solving it, then restarted with 10.1 by manually installing libtextcat, a listed dependency on http://wiki2.dovecot.org/Plugins/FTS/Lucene, but not part of the FreeBSD makefile: maybe that helped, maybe it was irrelevant).

However, I now get some anomalous results.  I'm putting them together as they may be related.  Or, maybe not.

1) Searching from Thunderbird returns 35x as many results for the same term as searching from the command line.  I'm not sure if the indexes are created or enumerable. 

2) There seems to be a glitch in the maildir naming structure that results in a double slash (//) which doesn't seem to impact IMAP serving or Sieve, but maybe is causing the problem above with Clucene.  Any hints?

If I run a search from Thunderbird and "run search on server" for a somewhat infrequently used term like "titanium," I get results in more or less every archive folder one by one over about 20 minutes of searching and get about 516 results.  It seems to run a little faster without the "run search on server" checkbox selected, but Thunderbird on Windows is abandonware and the only useful datapoint is 516 results.  Maybe it is time for Claws.

Anyway, if I run:

# doveadm search -u user at domain.com mailbox '*' body titanium
doveadm(user at domain.com): Error: stat(/mail/domain.com/user//.dovecot.sieve/tmp) failed: Not a directory
doveadm(user at domain.com): Error: Syncing mailbox dovecot.sieve failed: Internal error occurred. Refer to server log for more information. [2015-03-11 09:11:16]
bcadc922ba26b953a25701002812f29a 812478
bcadc922ba26b953a25701002812f29a 9338037
bcadc922ba26b953a25701002812f29a 9342918
bcadc922ba26b953a25701002812f29a 9343205
bcadc922ba26b953a25701002812f29a 9346096
bcadc922ba26b953a25701002812f29a 9350977
bcadc922ba26b953a25701002812f29a 9351379
bcadc922ba26b953a25701002812f29a 9352318
bcadc922ba26b953a25701002812f29a 9352441
bcadc922ba26b953a25701002812f29a 9352501
3d5d982e3a89b853c12c01002812f29a 1901940
3d5d982e3a89b853c12c01002812f29a 1908073
3d5d982e3a89b853c12c01002812f29a 1941453
3d5d982e3a89b853c12c01002812f29a 2036962
3d5d982e3a89b853c12c01002812f29a 2038360

I get 15 results.  

# doveadm fetch -u user at domain.com "mailbox" mailbox-guid 3d5d982e3a89b853c12c01002812f29a uid 1908073
mailbox: INBOX

All are from the inbox.

This takes roughly 30 seconds (much faster, that's good, though still not super fast) and in the logs I see promising messages like: 

Mar 11 09:27:52 host dovecot: indexer-worker(user at domain.com): Indexed 3708 messages in Commerce

but also slightly less promising messages like:

Mar 11 09:27:52 host dovecot: indexer-worker(user at domain.com): Warning: Maildir /mail/domain.com/user//.Bulkmail: Expunged message reappeared, giving a new UID (old uid=9335155, file=1426089637.M350915P91305.host,S=23845,W=24282:2,)

If I execute 
# doveadm index -A '*'

I get

doveadm(user at domain.com): Error: stat(/mail/domain.com/user//.dovecot.sieve/tmp) failed: Not a directory
doveadm(user at domain.com): Error: Syncing mailbox dovecot.sieve failed: Internal error occurred. Refer to server log for more information. [2015-03-11 09:51:26]

(there's nothing in the server log)  But note in both previous results a double slash (//) in the directory structure.  I don't think that's right.

So:  

Does # doveadm search search subfolders?  The internets say either '*' or INBOX should search the hierarchy and the search command does set off indexing operations on every subfolder as indicated by the logs, but the results are only from the top level folder....  perhaps that is related to...

Why do some errors report a double slash (//) in directory names?  I don't seem to be having any other symptoms (if, indeed, this is one) of incompatibilities.  My sieve filters work just fine and are accessible from the managesieve plugin in Thunderbird. 

My config file is below:

# 2.2.15: /usr/local/etc/dovecot/dovecot.conf
# Pigeonhole version 0.4.6 (3e924b1b6c5c+)
# OS: FreeBSD 10.1-RELEASE amd64
auth_default_realm = domain.com
auth_mechanisms = plain login
auth_realms = domain.com domain2.org
disable_plaintext_auth = no
first_valid_gid = 5000
first_valid_uid = 5000
hostname = domain.com
imap_idle_notify_interval = 29 mins
last_valid_gid = 5000
last_valid_uid = 5000
lda_mailbox_autocreate = yes
listen = *
mail_gid = 5000
mail_location = maildir:~:CONTROL=/var/no-quota/%u
mail_plugins = quota mail_log notify fts fts_lucene
mail_temp_dir = /var/tmp
mail_uid = 5000
maildir_broken_filename_sizes = yes
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave duplicate spamtest spamtestplus editheader
mbox_lazy_writes = no
namespace inbox {
  inbox = yes
  location =
  mailbox Drafts {
    special_use = \Drafts
  }
  mailbox Junk {
    special_use = \Junk
  }
  mailbox Sent {
    special_use = \Sent
  }
  mailbox "Sent Messages" {
    special_use = \Sent
  }
  mailbox Trash {
    special_use = \Trash
  }
  prefix =
}
passdb {
  args = /usr/local/etc/dovecot/dovecot-sql.conf.ext
  driver = sql
}
plugin {
  expire = Trash
  fts = lucene
  fts_autoindex = yes
  fts_lucene = whitespace_chars=@. mime_parts
  mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename
  mail_log_fields = uid box msgid size
  quota = maildir:User quota
  quota_exceeded_message = Storage quota for this account has been exceeded, please try again later.
  quota_rule = *:storage=50G
  quota_rule2 = Trash:storage=+30%%
  quota_rule3 = Sent:storage=+30%%
  quota_warning = storage=90%% quota-warning 90 %u
  quota_warning2 = storage=75%% quota-warning 75 %u
  sieve = ~/.dovecot.sieve
  sieve_before = /usr/local/etc/dovecot/sieve/
  sieve_dir = ~/sieve
  sieve_extensions = +spamtest +spamtestplus +relational +comparator-i;ascii-numeric +editheader
  sieve_global_dir = /usr/local/etc/dovecot/sieve
}
postmaster_address = user at domain.com
protocols = imap lmtp sieve sieve
sendmail_path = /usr/local/sbin/sendmail
service auth {
  unix_listener /var/spool/postfix/private/auth {
    group = postfix
    mode = 0666
    user = postfix
  }
  unix_listener auth-userdb {
    group = vmail
    mode = 0600
    user = vmail
  }
}
service managesieve-login {
  inet_listener sieve {
    port = 4190
  }
  service_count = 1
  vsz_limit = 128 M
}
service quota-warning {
  executable = script /usr/local/bin/quota-warning.sh
  unix_listener quota-warning {
    user = vmail
  }
  user = vmail
}
ssl = required
ssl_cert = </usr/local/etc/ca/host-cert.pem
ssl_key = </usr/local/etc/ca/host-unencrypted-key.pem
ssl_prefer_server_ciphers = yes
userdb {
  driver = prefetch
}
userdb {
  args = /usr/local/etc/dovecot/dovecot-sql.conf.ext
  driver = sql
}
protocol lda {
  mail_plugins = quota mail_log notify fts fts_lucene sieve
}
protocol imap {
  mail_max_userip_connections = 20
  mail_plugins = quota mail_log notify fts fts_lucene imap_quota imap_zlib
}
protocol lmtp {
  mail_plugins = quota mail_log notify fts fts_lucene sieve
}
(END)