[Dovecot] When are search indexes updated?

[Dave Abrahams]

on Tue Dec 04 2012, Timo Sirainen <tss-AT-iki.fi> wrote:

> On 4.12.2012, at 18.57, Dave Abrahams wrote:
>
>> Here's a further experience report with questions inline:
>> 
>> 1. "doveadm index '*'" crashes in clucene (for me), so it must be trying
>>   to update the FTS indexes, somehow.  Is that (the updating, not the
>>   crashing) intended behavior, and if so, should it be documented?
>
> doveadm index always triggers adding any unindexed messages to Dovecot
> index with mbox/Maildir code. With [sm]dbox all mails are always
> indexed, so this part doesn't do anything with them.
>
> When fts plugin is enabled, dovecot index also triggers the fts index
> updating of all messages not yet in the fts index. You usually should
> use the -q ("queue") parameter so that doveadm index only tells
> indexer process to start indexing. Without -q parameter the doveadm
> process itself does the indexing, but this is problematic with
> fts-lucene if another process attempts to index the mails at the same
> time. -q doesn't wait for indexer to finish.

Oh, this is hugely important information!  Wish I'd had that earlier.
Is that on the Wiki somewhere that I missed?

Huh, -q isn't even in the manpage for doveadm index.  Could you fix
that, please?

>> 2. "doveadm search text SOMETHINGthatWONTbeFOUND" takes a long time and
>>   finds stuff without crashing, but doesn't seem to create the
>>   lucene-indexes/ directory in my mdbox, and it takes a long time the
>>   next time around.  Is "doveadm search" intended to update the FTS
>>   indexes if they're missing?
>
> Yes. It should trigger the same doveadm index -q code. Maybe you have
> a (permission) problem connecting to indexer process. It should write
> about that to stderr.

Well, I'm not seeing anything like that :(

I just stopped dovecot, removed my lucene-indexes, started dovecot up
again, and issued "doveadm index -q / imap SEARCH body asdfg"
It came back immediately with a prompt, there was no output, and I don't
see an indexer-helper process.  In fact, the behavior is the same
without "-q"

>> 3. Performing a search on a large mailbox through IMAP takes a long time
>>   the first time around, during which "top" shows the big cpu hog is
>>   /opt/local/libexec/dovecot/indexer-worker, lucene-indexes/ is
>>   created, and subsequent searches go quickly.  Nice!  Is there a
>>   reasonably easy way to start such a search on all my mailboxes from
>>   the command line by using doveadm or the preauth tunnel?
>
> doveadm index -q / imap SEARCH body asdfg

This looks completely unlike anything I can find in "man doveadm-index";
are you sure it's right?  I can delete any number of arguments after the
"/" without seeming to affect its behavior.

-- 
Dave Abrahams
BoostPro Computing                  Software Development        Training
http://www.boostpro.com             Clang/LLVM/EDG Compilers  C++  Boost