Duplicate plugins - FTS Xapian

[Michael Slusarz]

See below.

> On 08/31/2021 5:35 AM Aki Tuomi <aki.tuomi at open-xchange.com> wrote:
> 
> > On 31/08/2021 13:59 Tom Hendrikx <tom at whyscream.net> wrote:
> >   
> > On 31-08-2021 12:01, Aki Tuomi wrote:
> > > 
> > >> On 31/08/2021 10:56 Felix Zielcke <fzielcke at z-51.de> wrote:
> > >>
> > >> Am Dienstag, dem 31.08.2021 um 10:33 +0300 schrieb Aki Tuomi:
> > >>>
> > >>>> On 31/08/2021 00:11 Joan Moreau <jom at grosjo.net> wrote:
> > >>>>
> > >>>> Hi
> > >>>> There seems to be 2 plugins doing the same thins
> > >>>> - https://github.com/slusarz/dovecot-fts-flatcurve/
> > >>>> - https://github.com/grosjo/fts-xapian/ (mine)

...

> > >>>> Isn't there double work here ?

FYI, Joan also reached out to me on the Github project page, so you can see my response to this question there:

https://github.com/slusarz/dovecot-fts-flatcurve/issues/5

Short answer: I see the plugins as "different".  Yes, they both use the same underlying indexing technology (Xapian), but the design goals, features, and implementations are unique for each project.  Thus, I don't see them as duplicate work.


> > >> Is there somewhere a direct comparison of them?
> > >> I currenty use fts-xapian from Joan without problems.
> > >> But what would be the advantages of fts-flatcurve over fts-xapian?

I was asked a similar question previously.  See https://github.com/slusarz/dovecot-fts-flatcurve/issues/4#issuecomment-902425597


> > > fts_flatcurve does only full word searching, although you can use fts_filters and fts_tokenizers settings to affect stemming and other matching to make it work with plurals and such.

Aki's knowledge is a bit dated - fts_flatcurve now supports both prefix matching (this is what Xapian provides by default) AND full substring matching (this is what is technically required by the IMAP RFC).

Although the current fts_flatcurve default is for full substring matching, it is debatable whether that is the correct choice.  Users tend to think of search in terms of "prefix" matching - that's how Google works - and substring matching requires significant additional storage since Xapian doesn't support it natively.


> > I still think it's weird to see that Open-Xchange starts a FTS Xapian 
> > plugin with mostly the same basic functionality that is already 
> > available in an existing plugin maintained by someone in the community 

As Aki noted, Open-Xchange didn't "start" anything.  fts_flatcurve is entirely a project I have developed on my personal time.

As a Dovecot CE user, I had the same complaint that most people have - in current versions of Dovecot, there is no easy-to-use FTS option that works out-of-the-box.  Solr is the only search driver still actively supported, and it requires significant additional resources and configuration to setup.

Unfortunately, we don't have the resources within Open-Xchange to create a CE-only search option as our customers generally use our proprietary, object storage aware FTS system that is part of OX Dovecot Pro.  Without commercial demand, it is difficult to justify paid work on features that do not produce revenue in return.  (fts_flatcurve does use Dovecot core features that are maintained for the Pro FTS system, like stemming, so we can take advantage of trickle down benefits from the commercial code maintenance.)

My goal was to give back to the community by providing an easy-to-use, resource sensitive, no-maintenance option to address this current product limitation.

Hopefully, fts_flatcurve will fit the requirements for most smaller installations that run Dovecot that don't require the carrier/enterprise grade FTS solutions large customers demand.


> > Especially if that happens without any (apparent) communication with the 
> > existing plugin developer to find out whether fixing the issues that 
> > slusarz/Open-Xchange seem to have with the existing plugin, can be fixed.

There were fundamental disagreements about design goals, and Joan (or anybody, really) would likely not have appreciated someone coming in and demanding that the entire architecture and design of the project needed to be changed based on my personal opinions.

This is open source.  Everybody should be free to participate as they wish, as long as you are not negatively affecting someone else.  I don't think that has happened here.


> > Combining forces just seems a better way to spend scarce development 
> > resources than building something similar (but different) without any 
> > communication.

I disagree, but that's ok.  People can do that.

michael