[Dovecot] xapian search plugin

[Timo Sirainen]

On Thu, 2010-12-16 at 13:00 +0300, Vasiliy G Tolstov wrote:
> Hello. Does anybody write fts xapian plugin to dovecot 2? 

Not as far as I know.

> If nobody -
> can You suggest me what i need to read and what source files i need no
> read to write own xapian fts plugin.

src/plugins/fts/fts-api*.h
src/plugins/fts-solr/*
http://wiki2.dovecot.org/Design

There are some stupid things in the FTS API, like having lookup(),
lookup2() and filter() methods. I don't remember clearly what the point
was, but just do it like fts-solr does :)

There are some things you maybe shouldn't do like Solr though. There are
two things in my TODO list:

1. fts-solr: crashes if expunge is done while search is indexing
2. fts-solr: handle DELETE, RENAME. use mailbox GUIDs (optionally)

The first is caused because Dovecot supports processing multiple
commands in parallel. So if a client issues a SEARCH that starts
indexing messages, and then while the indexing is going on it issues
EXPUNGE command, the Solr code's expunge() method is called, but that
breaks because the search indexing is still going on and it can't handle
that. I haven't yet decided what to do about that. It's a bit rare
problem though.

The second part then has two issues. Not handling DELETE at all is
probably FTS API's problem, it should probably just be issuing expunge()
for all of the messages in mailbox while it's being deleted.

The RENAME is also kind of annoying, because renaming a mailbox requires
all messages in it to be reindexed (Solr doesn't support modifying
existing fields). I was thinking that perhaps instead of using mailbox
names in the FTS indexes, it could use mailbox global UIDs. The GUIDs
never change, so RENAME doesn't require any action. Filtering based on
mailbox names would get a bit trickier then though, because they would
always have to be first translated to GUIDs.

Yet another possibility would be to not save mailbox name or GUID at
all, just store message GUIDs. This would make it possible that COPYing
a message to another mailbox wouldn't require reindexing it. The main
problem then is of course that you'd need to have GUID -> array of
{mailbox, uid} index somewhere, and Dovecot doesn't yet have one.