Hi,

I installed and tested your version, but the indexer process crashes reproducible with the following command after about 2000 messages were indexed:

doveadm index -u paul@iwascoding.com -q \*

Jan 14 09:26:15 mail dovecot: indexer-worker(paul@iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Mailbox sent: UID 2038: Opened mail because: fts indexing
Jan 14 09:26:15 mail dovecot: indexer-worker: Error: terminate called after throwing an instance of 'std::bad_alloc'
Jan 14 09:26:15 mail dovecot: indexer-worker: Error: what(): std::bad_alloc
Jan 14 09:26:15 mail dovecot: indexer: Error: Indexer worker disconnected, discarding 48 requests for paul@iwascoding.com
Jan 14 09:26:15 mail dovecot: indexer-worker(paul@iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Fatal: master: service(indexer-worker): child 16777 killed with signal 6 (core dumps disabled - https://dovecot.org/bugreport.html#coredumps)

I tried to delete the message, but this does not help (crashes e.g. after message 2029 or 2044). Other folders with fewer messages were successfully indexed before.

Sorry, could not convince dovecot to create core dumps (read the docs, changed /proc/sys/kernel/core_pattern, added LimitCORE=unlimited/infinity, even created /etc/systemd/system/dovecot.service.d/coredump.conf to no avail). Custom Dovecot 2.3.4 on Debian Stretch.

Thanks,
Paul

On 14. Jan 2019, at 07:42, Joan Moreau via dovecot <dovecot@dovecot.org> wrote:

Thank you Stephan.

The version here shall be up and running : https://github.com/grosjo/fts-xapian

On 2019-01-14 00:07, Stephan Bosch wrote:

Op 13/01/2019 om 21:25 schreef Joan Moreau via dovecot:

I tried to combined it, the "autoreconf" errors are solved

Now, when I type "make install", the lib is not pushed into dovecot folder, but somewhere in /usr/local/...

How to adjust this to have it arriving in the proper folder ?

Depends on your system. It mostly a matter of setting a proper --prefix directory for configure, but other paths are configurable as well. I usually check what the official distribution package for Dovecot is doing and use that as a basis.

For Debian I use the following configure command:

./configure --with-ldap=plugin --with-ssl=openssl --with-sql=plugin --with-lua=plugin --with-pgsql --with-mysql --with-sqlite \
        --with-gssapi=plugin --with-solr --with-ioloop=best --enable-maintainer-mode \
        --prefix=/usr --sysconfdir=/etc --libexecdir=/usr/lib --localstatedir=/var --mandir=/usr/share/man \
        --infodir=/usr/share/info --with-moduledir=/usr/lib/dovecot/modules --disable-rpath --disable-static

Regards,

Stephan

On 2019-01-13 21:01, Tuomi, Aki wrote:

You copied your Makefile.am there. Stephan made you a working version, can you try that?
(sorry for dup)
Aki
-------- Original message --------
From: Joan Moreau <jom@grosjo.net>
Date: 13/01/2019 21:39 (GMT+02:00)
To: Stephan Bosch <stephan@rename-it.nl>
Cc: Aki Tuomi <aki.tuomi@open-xchange.com>
Subject: Re: [FTS Xapian] Beta release

I used the skeleton from Aki : https://github.com/grosjo/fts-xapian

However, when I try to act as a visitor, I reach teh follwoing error:

# autoreconf -vi
autoreconf: Entering directory `.'
autoreconf: configure.ac: not using Gettext
autoreconf: running: aclocal -I m4
autoreconf: configure.ac: tracing
autoreconf: running: libtoolize --copy
libtoolize: putting auxiliary files in AC_CONFIG_AUX_DIR, '.'.
libtoolize: copying file './ltmain.sh'
libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'm4'.
libtoolize: copying file 'm4/libtool.m4'
libtoolize: copying file 'm4/ltoptions.m4'
libtoolize: copying file 'm4/ltsugar.m4'
libtoolize: copying file 'm4/ltversion.m4'
libtoolize: copying file 'm4/lt~obsolete.m4'
autoreconf: running: /usr/bin/autoconf
autoreconf: running: /usr/bin/autoheader
autoreconf: running: automake --add-missing --copy --no-force
configure.ac:9: installing './compile'
configure.ac:11: installing './config.guess'
configure.ac:11: installing './config.sub'
configure.ac:7: installing './install-sh'
configure.ac:7: installing './missing'
src/Makefile.am: installing './depcomp'
/usr/share/automake-1.16/am/depend2.am: error: am__fastdepCXX does not appear in AM_CONDITIONAL
/usr/share/automake-1.16/am/depend2.am: The usual way to define 'am__fastdepCXX' is to add 'AC_PROG_CXX'
/usr/share/automake-1.16/am/depend2.am: to 'configure.ac' and run 'aclocal' and 'autoconf' again
src/Makefile.am: error: C++ source seen but 'CXX' is undefined
src/Makefile.am: The usual way to define 'CXX' is to add 'AC_PROG_CXX'
src/Makefile.am: to 'configure.ac' and run 'autoconf' again.
src/Makefile.am:11: warning: variable 'NOPLUGIN_LDFLAGS' is defined but no program or
src/Makefile.am:11: library has 'NOPLUGIN' as canonical name (possible typo)
autoreconf: automake failed with exit status: 1

On 2019-01-13 20:24, Stephan Bosch wrote:

    Oh, right, a distribution tarball doesn't include some of the
    necessary files for your repository like autogen.sh and
    .gitignore. The attached tarball includes all those and is ready
    for `git init`. The previous tarball was made with `make
    distcheck` from this one.

    Regards,

    Stephan.

    Op 13/01/2019 om 20:14 schreef Stephan Bosch:

        Hi Joan,

        Op 13/01/2019 om 19:03 schreef Aki Tuomi:

            Yes, from compiling point of view it is done.

            Unfortunately what is not done is all the other work
            involved, such as fixing all the inevitable bugs it has
            and maintaining it. We do not want, at this moment, take
            up maintaining and developing yet another FTS plugin as
            we have plenty of things to do already.

            I invite you to setup your own repository and provide
            this plugin from there, being the maintainer of this
            plugin. We can add a link to your plugin on our FTS page
            so people can also find it.

            There are other plugins like this, e.g.
            https://github.com/st3fan/dovecot-xaps-plugin

        I turned the code you provided into a separate plugin
        package. The distribution tarball is attached.

        Notable changes:

        - Added example copyright headers and COPYING and AUTHORS
        files. You should modify those to your preference.
        - Added README and INSTALL files (in markdown using Pandoc).
        Those need to be amended with details.
        - Amended the plugin code to display a debug message with the
        plugin name and version upon plugin load.

        I advise you to turn this into a git repository and continue from there.

        I do not recommend releasing this plugin with the
        -fpermissive flag and the resulting warning as it is now. But
        I'm assuming this is still a work in progress, so that is OK.

        Regards,

        Stephan.

                On 13 January 2019 at 19:52 Joan Moreau <jom@grosjo.net
                <mailto:jom@grosjo.net>> wrote:

                The only point here of this fts-xapian is to get rid
                of solr (because it
                is just a nightmare to setup) and squat (because it is considere
                obsolete).

                I already sent the changed in configure.ac, makefile.am, etc.. in order
                to include it in the dovecot, and it compiles properly

                The only remaining point is to push it in hte git (yes, everything is
                already done)

                On 2019-01-13 18:45, Aki Tuomi wrote:

                        On 13 January 2019 at 17:05 Joan Moreau via
                        dovecot <dovecot@dovecot.org
                        <mailto:dovecot@dovecot.org>> wrote:

                        Hi

                        Please find attached the beta release of FTS
                        Xapian, with the objective
                        to replace fts_squat that is being deprecated.

                        Configuration is exactly the same as for fts_squat:

                        plugin {

                        plugin = fts fts_xapian (...)
                        fts = xapian
                        fts_autoindex = yes
                        fts_enforced = yes
                        fts_xapian = partial=2 full=20

                        This is installed on my production server
                        (>120Gb of mailboxes), and I
                        will observe it during the coming days.

                        I will definitely appreciate that this is added in the core git of
                        docevot, in order to have a versionning of
                        it, to remove squat and let
                        basic users able to avoid Solr alternative as much as possible.

                        Thanks

                        JM

                    Hi!

                    I still recommend you setup a, say, github
                    repository for your plugin. We are not able to
                    currently include your work in dovecot core as it
                    is more work than just pushing the code into the
                    repo. Maybe it can be included in the future.

                    If you want, I can help you in setting up the
                    required configuration scripts and such to make
                    it possible to compile it as plugin.

                    Then anyone can download it and install it for
                    their dovecot, even if dovecot itself has been
                    installed from packages, and also makes it
                    possible for package maintainers to consider
                    including it in distributions.

                    Aki

Paul Hecker
paul@iwascoding.com
http://www.iwascoding.com

iwascoding GmbH
Weserstr 18, 10247 Berlin
HRB 97269 B Amtsgericht Berlin-Charlottenburg
Geschäftsführer: Paul Hecker, Ilja Iwas