I recently got FTS search with Clucene working on FreeBSD 10.1 (I had some trouble with 9.3, postponed solving it, then restarted with 10.1 by manually installing libtextcat, a listed dependency on http://wiki2.dovecot.org/Plugins/FTS/Lucene, but not part of the FreeBSD makefile: maybe that helped, maybe it was irrelevant).

However, I now get some anomalous results. I'm putting them together as they may be related. Or, maybe not.

1. Searching from Thunderbird returns 35x as many results for the same term as searching from the command line. I'm not sure if the indexes are created or enumerable.

2. There seems to be a glitch in the maildir naming structure that results in a double slash (//) which doesn't seem to impact IMAP serving or Sieve, but maybe is causing the problem above with Clucene. Any hints?

If I run a search from Thunderbird and "run search on server" for a somewhat infrequently used term like "titanium," I get results in more or less every archive folder one by one over about 20 minutes of searching and get about 516 results. It seems to run a little faster without the "run search on server" checkbox selected, but Thunderbird on Windows is abandonware and the only useful datapoint is 516 results. Maybe it is time for Claws.

Anyway, if I run:

# doveadm search -u user@domain.com mailbox '*' body titanium doveadm(user@domain.com): Error: stat(/mail/domain.com/user//.dovecot.sieve/tmp) failed: Not a directory doveadm(user@domain.com): Error: Syncing mailbox dovecot.sieve failed: Internal error occurred. Refer to server log for more information. [2015-03-11 09:11:16] bcadc922ba26b953a25701002812f29a 812478 bcadc922ba26b953a25701002812f29a 9338037 bcadc922ba26b953a25701002812f29a 9342918 bcadc922ba26b953a25701002812f29a 9343205 bcadc922ba26b953a25701002812f29a 9346096 bcadc922ba26b953a25701002812f29a 9350977 bcadc922ba26b953a25701002812f29a 9351379 bcadc922ba26b953a25701002812f29a 9352318 bcadc922ba26b953a25701002812f29a 9352441 bcadc922ba26b953a25701002812f29a 9352501 3d5d982e3a89b853c12c01002812f29a 1901940 3d5d982e3a89b853c12c01002812f29a 1908073 3d5d982e3a89b853c12c01002812f29a 1941453 3d5d982e3a89b853c12c01002812f29a 2036962 3d5d982e3a89b853c12c01002812f29a 2038360

I get 15 results.

# doveadm fetch -u user@domain.com "mailbox" mailbox-guid 3d5d982e3a89b853c12c01002812f29a uid 1908073 mailbox: INBOX

All are from the inbox.

This takes roughly 30 seconds (much faster, that's good, though still not super fast) and in the logs I see promising messages like:

Mar 11 09:27:52 host dovecot: indexer-worker(user@domain.com): Indexed 3708 messages in Commerce

but also slightly less promising messages like:

Mar 11 09:27:52 host dovecot: indexer-worker(user@domain.com): Warning: Maildir /mail/domain.com/user//.Bulkmail: Expunged message reappeared, giving a new UID (old uid=9335155, file=1426089637.M350915P91305.host,S=23845,W=24282:2,)

If I execute # doveadm index -A '*'

I get

doveadm(user@domain.com): Error: stat(/mail/domain.com/user//.dovecot.sieve/tmp) failed: Not a directory doveadm(user@domain.com): Error: Syncing mailbox dovecot.sieve failed: Internal error occurred. Refer to server log for more information. [2015-03-11 09:51:26]

(there's nothing in the server log) But note in both previous results a double slash (//) in the directory structure. I don't think that's right.

So:

Does # doveadm search search subfolders? The internets say either '*' or INBOX should search the hierarchy and the search command does set off indexing operations on every subfolder as indicated by the logs, but the results are only from the top level folder.... perhaps that is related to...

Why do some errors report a double slash (//) in directory names? I don't seem to be having any other symptoms (if, indeed, this is one) of incompatibilities. My sieve filters work just fine and are accessible from the managesieve plugin in Thunderbird.

My config file is below:

# 2.2.15: /usr/local/etc/dovecot/dovecot.conf # Pigeonhole version 0.4.6 (3e924b1b6c5c+) # OS: FreeBSD 10.1-RELEASE amd64 auth_default_realm = domain.com auth_mechanisms = plain login auth_realms = domain.com domain2.org disable_plaintext_auth = no first_valid_gid = 5000 first_valid_uid = 5000 hostname = domain.com imap_idle_notify_interval = 29 mins last_valid_gid = 5000 last_valid_uid = 5000 lda_mailbox_autocreate = yes listen = * mail_gid = 5000 mail_location = maildir:~:CONTROL=/var/no-quota/%u mail_plugins = quota mail_log notify fts fts_lucene mail_temp_dir = /var/tmp mail_uid = 5000 maildir_broken_filename_sizes = yes managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave duplicate spamtest spamtestplus editheader mbox_lazy_writes = no namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { args = /usr/local/etc/dovecot/dovecot-sql.conf.ext driver = sql } plugin { expire = Trash fts = lucene fts_autoindex = yes fts_lucene = whitespace_chars=@. mime_parts mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename mail_log_fields = uid box msgid size quota = maildir:User quota quota_exceeded_message = Storage quota for this account has been exceeded, please try again later. quota_rule = *:storage=50G quota_rule2 = Trash:storage=+30%% quota_rule3 = Sent:storage=+30%% quota_warning = storage=90%% quota-warning 90 %u quota_warning2 = storage=75%% quota-warning 75 %u sieve = ~/.dovecot.sieve sieve_before = /usr/local/etc/dovecot/sieve/ sieve_dir = ~/sieve sieve_extensions = +spamtest +spamtestplus +relational +comparator-i;ascii-numeric +editheader sieve_global_dir = /usr/local/etc/dovecot/sieve } postmaster_address = user@domain.com protocols = imap lmtp sieve sieve sendmail_path = /usr/local/sbin/sendmail service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0666 user = postfix } unix_listener auth-userdb { group = vmail mode = 0600 user = vmail } } service managesieve-login { inet_listener sieve { port = 4190 } service_count = 1 vsz_limit = 128 M } service quota-warning { executable = script /usr/local/bin/quota-warning.sh unix_listener quota-warning { user = vmail } user = vmail } ssl = required ssl_cert = </usr/local/etc/ca/host-cert.pem ssl_key = </usr/local/etc/ca/host-unencrypted-key.pem ssl_prefer_server_ciphers = yes userdb { driver = prefetch } userdb { args = /usr/local/etc/dovecot/dovecot-sql.conf.ext driver = sql } protocol lda { mail_plugins = quota mail_log notify fts fts_lucene sieve } protocol imap { mail_max_userip_connections = 20 mail_plugins = quota mail_log notify fts fts_lucene imap_quota imap_zlib } protocol lmtp { mail_plugins = quota mail_log notify fts fts_lucene sieve } (END)