Op 13/01/2019 om 21:25 schreef Joan Moreau via dovecot:

I tried to combined it, the "autoreconf" errors are solved

Now, when I type "make install", the lib is not pushed into dovecot folder, but somewhere in /usr/local/...

How to adjust this to have it arriving in the proper folder ?

Depends on your system. It mostly a matter of setting a proper --prefix directory for configure, but other paths are configurable as well. I usually check what the official distribution package for Dovecot is doing and use that as a basis.

For Debian I use the following configure command:

./configure --with-ldap=plugin --with-ssl=openssl --with-sql=plugin --with-lua=plugin --with-pgsql --with-mysql --with-sqlite
--with-gssapi=plugin --with-solr --with-ioloop=best --enable-maintainer-mode
--prefix=/usr --sysconfdir=/etc --libexecdir=/usr/lib --localstatedir=/var --mandir=/usr/share/man
--infodir=/usr/share/info --with-moduledir=/usr/lib/dovecot/modules --disable-rpath --disable-static

Regards,

Stephan

On 2019-01-13 21:01, Tuomi, Aki wrote:

...

You copied your Makefile.am there. Stephan made you a working version, can you try that? (sorry for dup) Aki -------- Original message -------- From: Joan Moreau jom@grosjo.net Date: 13/01/2019 21:39 (GMT+02:00) To: Stephan Bosch stephan@rename-it.nl Cc: Aki Tuomi aki.tuomi@open-xchange.com Subject: Re: [FTS Xapian] Beta release

I used the skeleton from Aki : https://github.com/grosjo/fts-xapian

However, when I try to act as a visitor, I reach teh follwoing error:

# autoreconf -vi autoreconf: Entering directory `.' autoreconf: configure.ac: not using Gettext autoreconf: running: aclocal -I m4 autoreconf: configure.ac: tracing autoreconf: running: libtoolize --copy libtoolize: putting auxiliary files in AC_CONFIG_AUX_DIR, '.'. libtoolize: copying file './ltmain.sh' libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'm4'. libtoolize: copying file 'm4/libtool.m4' libtoolize: copying file 'm4/ltoptions.m4' libtoolize: copying file 'm4/ltsugar.m4' libtoolize: copying file 'm4/ltversion.m4' libtoolize: copying file 'm4/lt~obsolete.m4' autoreconf: running: /usr/bin/autoconf autoreconf: running: /usr/bin/autoheader autoreconf: running: automake --add-missing --copy --no-force configure.ac:9: installing './compile' configure.ac:11: installing './config.guess' configure.ac:11: installing './config.sub' configure.ac:7: installing './install-sh' configure.ac:7: installing './missing' src/Makefile.am: installing './depcomp' /usr/share/automake-1.16/am/depend2.am: error: am__fastdepCXX does not appear in AM_CONDITIONAL /usr/share/automake-1.16/am/depend2.am: The usual way to define 'am__fastdepCXX' is to add 'AC_PROG_CXX' /usr/share/automake-1.16/am/depend2.am: to 'configure.ac' and run 'aclocal' and 'autoconf' again src/Makefile.am: error: C++ source seen but 'CXX' is undefined src/Makefile.am: The usual way to define 'CXX' is to add 'AC_PROG_CXX' src/Makefile.am: to 'configure.ac' and run 'autoconf' again. src/Makefile.am:11: warning: variable 'NOPLUGIN_LDFLAGS' is defined but no program or src/Makefile.am:11: library has 'NOPLUGIN' as canonical name (possible typo) autoreconf: automake failed with exit status: 1

On 2019-01-13 20:24, Stephan Bosch wrote:

Oh, right, a distribution tarball doesn't include some of the
necessary files for your repository like autogen.sh and
.gitignore. The attached tarball includes all those and is ready
for `git init`. The previous tarball was made with `make
distcheck` from this one.

Regards,

Stephan.

Op 13/01/2019 om 20:14 schreef Stephan Bosch:

    Hi Joan,

    Op 13/01/2019 om 19:03 schreef Aki Tuomi:

        Yes, from compiling point of view it is done.

        Unfortunately what is not done is all the other work
        involved, such as fixing all the inevitable bugs it has
        and maintaining it. We do not want, at this moment, take
        up maintaining and developing yet another FTS plugin as
        we have plenty of things to do already.

        I invite you to setup your own repository and provide
        this plugin from there, being the maintainer of this
        plugin. We can add a link to your plugin on our FTS page
        so people can also find it.

        There are other plugins like this, e.g.
        https://github.com/st3fan/dovecot-xaps-plugin


    I turned the code you provided into a separate plugin
    package. The distribution tarball is attached.

    Notable changes:

    - Added example copyright headers and COPYING and AUTHORS
    files. You should modify those to your preference.
    - Added README and INSTALL files (in markdown using Pandoc).
    Those need to be amended with details.
    - Amended the plugin code to display a debug message with the
    plugin name and version upon plugin load.

    I advise you to turn this into a git repository and continue from there.

    I do not recommend releasing this plugin with the
    -fpermissive flag and the resulting warning as it is now. But
    I'm assuming this is still a work in progress, so that is OK.

    Regards,

    Stephan.


            On 13 January 2019 at 19:52 Joan Moreau <jom@grosjo.net
            <mailto:jom@grosjo.net>> wrote:


            The only point here of this fts-xapian is to get rid
            of solr (because it
            is just a nightmare to setup) and squat (because it is considere
            obsolete).

            I already sent the changed in configure.ac, makefile.am, etc.. in order
            to include it in the dovecot, and it compiles properly

            The only remaining point is to push it in hte git (yes, everything is
            already done)

            On 2019-01-13 18:45, Aki Tuomi wrote:

                    On 13 January 2019 at 17:05 Joan Moreau via
                    dovecot <dovecot@dovecot.org
                    <mailto:dovecot@dovecot.org>> wrote:

                    Hi

                    Please find attached the beta release of FTS
                    Xapian, with the objective
                    to replace fts_squat that is being deprecated.

                    Configuration is exactly the same as for fts_squat:

                    plugin {

                    plugin = fts fts_xapian (...)
                    fts = xapian
                    fts_autoindex = yes
                    fts_enforced = yes
                    fts_xapian = partial=2 full=20

                    This is installed on my production server
                    (>120Gb of mailboxes), and I
                    will observe it during the coming days.

                    I will definitely appreciate that this is added in the core git of
                    docevot, in order to have a versionning of
                    it, to remove squat and let
                    basic users able to avoid Solr alternative as much as possible.

                    Thanks

                    JM

                Hi!

                I still recommend you setup a, say, github
                repository for your plugin. We are not able to
                currently include your work in dovecot core as it
                is more work than just pushing the code into the
                repo. Maybe it can be included in the future.

                If you want, I can help you in setting up the
                required configuration scripts and such to make
                it possible to compile it as plugin.

                Then anyone can download it and install it for
                their dovecot, even if dovecot itself has been
                installed from packages, and also makes it
                possible for package maintainers to consider
                including it in distributions.

                Aki

In your README.md, perhaps "This project intends to provide a straightforward and simple *procedure *to configure FTS plugin for Dovecot, leveraging the efforts by the Xapian.org team." is better?? Also in the part after cloning from git:

./configure --prefix=/usr --with-dovecot=/path/to/dovecot [ This /path/to/dovecot is not obvious. Is it the dovecot binary or what??]

On Mon, 14 Jan 2019 at 09:42, Joan Moreau via dovecot dovecot@dovecot.org wrote:

Thank you Stephan.

The version here shall be up and running : https://github.com/grosjo/fts-xapian

On 2019-01-14 00:07, Stephan Bosch wrote:

Op 13/01/2019 om 21:25 schreef Joan Moreau via dovecot:

I tried to combined it, the "autoreconf" errors are solved

Now, when I type "make install", the lib is not pushed into dovecot folder, but somewhere in /usr/local/...

How to adjust this to have it arriving in the proper folder ?

Depends on your system. It mostly a matter of setting a proper --prefix directory for configure, but other paths are configurable as well. I usually check what the official distribution package for Dovecot is doing and use that as a basis.

For Debian I use the following configure command:

./configure --with-ldap=plugin --with-ssl=openssl --with-sql=plugin --with-lua=plugin --with-pgsql --with-mysql --with-sqlite
--with-gssapi=plugin --with-solr --with-ioloop=best --enable-maintainer-mode
--prefix=/usr --sysconfdir=/etc --libexecdir=/usr/lib --localstatedir=/var --mandir=/usr/share/man
--infodir=/usr/share/info --with-moduledir=/usr/lib/dovecot/modules --disable-rpath --disable-static

Regards,

Stephan

On 2019-01-13 21:01, Tuomi, Aki wrote:

You copied your Makefile.am there. Stephan made you a working version, can you try that? (sorry for dup) Aki -------- Original message -------- From: Joan Moreau jom@grosjo.net Date: 13/01/2019 21:39 (GMT+02:00) To: Stephan Bosch stephan@rename-it.nl Cc: Aki Tuomi aki.tuomi@open-xchange.com Subject: Re: [FTS Xapian] Beta release

I used the skeleton from Aki : https://github.com/grosjo/fts-xapian

However, when I try to act as a visitor, I reach teh follwoing error:

# autoreconf -vi autoreconf: Entering directory `.' autoreconf: configure.ac: not using Gettext autoreconf: running: aclocal -I m4 autoreconf: configure.ac: tracing autoreconf: running: libtoolize --copy libtoolize: putting auxiliary files in AC_CONFIG_AUX_DIR, '.'. libtoolize: copying file './ltmain.sh' libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'm4'. libtoolize: copying file 'm4/libtool.m4' libtoolize: copying file 'm4/ltoptions.m4' libtoolize: copying file 'm4/ltsugar.m4' libtoolize: copying file 'm4/ltversion.m4' libtoolize: copying file 'm4/lt~obsolete.m4' autoreconf: running: /usr/bin/autoconf autoreconf: running: /usr/bin/autoheader autoreconf: running: automake --add-missing --copy --no-force configure.ac:9: installing './compile' configure.ac:11: installing './config.guess' configure.ac:11: installing './config.sub' configure.ac:7: installing './install-sh' configure.ac:7: installing './missing' src/Makefile.am: installing './depcomp' /usr/share/automake-1.16/am/depend2.am: error: am__fastdepCXX does not appear in AM_CONDITIONAL /usr/share/automake-1.16/am/depend2.am: The usual way to define 'am__fastdepCXX' is to add 'AC_PROG_CXX' /usr/share/automake-1.16/am/depend2.am: to 'configure.ac' and run 'aclocal' and 'autoconf' again src/Makefile.am: error: C++ source seen but 'CXX' is undefined src/Makefile.am: The usual way to define 'CXX' is to add 'AC_PROG_CXX' src/Makefile.am: to 'configure.ac' and run 'autoconf' again. src/Makefile.am:11: warning: variable 'NOPLUGIN_LDFLAGS' is defined but no program or src/Makefile.am:11: library has 'NOPLUGIN' as canonical name (possible typo) autoreconf: automake failed with exit status: 1

On 2019-01-13 20:24, Stephan Bosch wrote:

Oh, right, a distribution tarball doesn't include some of the
necessary files for your repository like autogen.sh and
.gitignore. The attached tarball includes all those and is ready
for `git init`. The previous tarball was made with `make
distcheck` from this one.

Regards,

Stephan.

Op 13/01/2019 om 20:14 schreef Stephan Bosch:

    Hi Joan,

    Op 13/01/2019 om 19:03 schreef Aki Tuomi:

        Yes, from compiling point of view it is done.

        Unfortunately what is not done is all the other work
        involved, such as fixing all the inevitable bugs it has
        and maintaining it. We do not want, at this moment, take
        up maintaining and developing yet another FTS plugin as
        we have plenty of things to do already.

        I invite you to setup your own repository and provide
        this plugin from there, being the maintainer of this
        plugin. We can add a link to your plugin on our FTS page
        so people can also find it.

        There are other plugins like this, e.g.
        https://github.com/st3fan/dovecot-xaps-plugin


    I turned the code you provided into a separate plugin
    package. The distribution tarball is attached.

    Notable changes:

    - Added example copyright headers and COPYING and AUTHORS
    files. You should modify those to your preference.
    - Added README and INSTALL files (in markdown using Pandoc).
    Those need to be amended with details.
    - Amended the plugin code to display a debug message with the
    plugin name and version upon plugin load.


    I advise you to turn this into a git repository and continue from there.

    I do not recommend releasing this plugin with the
    -fpermissive flag and the resulting warning as it is now. But
    I'm assuming this is still a work in progress, so that is OK.

    Regards,

    Stephan.


            On 13 January 2019 at 19:52 Joan Moreau <jom@grosjo.net
            <mailto:jom@grosjo.net>> wrote:


            The only point here of this fts-xapian is to get rid
            of solr (because it

            is just a nightmare to setup) and squat (because it is considere
            obsolete).

            I already sent the changed in configure.ac, makefile.am

, etc.. in order to include it in the dovecot, and it compiles properly

            The only remaining point is to push it in hte git (yes, everything is
            already done)

            On 2019-01-13 18:45, Aki Tuomi wrote:

                    On 13 January 2019 at 17:05 Joan Moreau via
                    dovecot <dovecot@dovecot.org
                    <mailto:dovecot@dovecot.org>> wrote:

                    Hi

                    Please find attached the beta release of FTS
                    Xapian, with the objective
                    to replace fts_squat that is being deprecated.

                    Configuration is exactly the same as for fts_squat:

                    plugin {

                    plugin = fts fts_xapian (...)
                    fts = xapian
                    fts_autoindex = yes
                    fts_enforced = yes
                    fts_xapian = partial=2 full=20

                    This is installed on my production server
                    (>120Gb of mailboxes), and I
                    will observe it during the coming days.


                    I will definitely appreciate that this is added in the core git of
                    docevot, in order to have a versionning of
                    it, to remove squat and let

                    basic users able to avoid Solr alternative as much as possible.

                    Thanks

                    JM

                Hi!

                I still recommend you setup a, say, github
                repository for your plugin. We are not able to
                currently include your work in dovecot core as it
                is more work than just pushing the code into the
                repo. Maybe it can be included in the future.

                If you want, I can help you in setting up the
                required configuration scripts and such to make
                it possible to compile it as plugin.

                Then anyone can download it and install it for
                their dovecot, even if dovecot itself has been
                installed from packages, and also makes it
                possible for package maintainers to consider
                including it in distributions.

                Aki

-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 "Oh, the cruft.", grep ^[^#] :-)

Testing a compile on FreeBSD.

gmake[2]: Entering directory '/usr/home/wash/Tools/Dovecot/fts-xapian/src' /bin/sh ../libtool --tag=CXX --mode=compile c++ -DHAVE_CONFIG_H -I. -I.. -I/opt/dovecot2.3/include/dovecot -I/opt/dovecot2.3/include/dovecot -g -O2 -MT fts-backend-xapian.lo -MD -MP -MF .deps/fts-backend-xapian.Tpo -c -o fts-backend-xapian.lo fts-backend-xapian.cpp libtool: compile: c++ -DHAVE_CONFIG_H -I. -I.. -I/opt/dovecot2.3/include/dovecot -I/opt/dovecot2.3/include/dovecot -g -O2 -MT fts-backend-xapian.lo -MD -MP -MF .deps/fts-backend-xapian.Tpo -c fts-backend-xapian.cpp -fPIC -DPIC -o .libs/fts-backend-xapian.o fts-backend-xapian.cpp:3:10: fatal error: 'xapian.h' file not found #include ^~~~~~~~~~

Well, I installed xapian-core and the xapian.h is in /usr/local/include/

I can overcome the fatal error by doing:

env CPPFLAGS=-I/usr/local/include PANDOC=false ./configure --prefix=/opt --with-dovecot=/opt/dovecot2.3/lib/dovecot/

Is that something that you can address within the code or we (*BSD) have to live with it?

During make, the following warning is generated:

/bin/sh ../libtool --tag=CXX --mode=compile c++ -DHAVE_CONFIG_H -I. -I.. -I/opt/dovecot2.3/include/dovecot -I/usr/local/include -I/opt/dovecot2.3/include/dovecot -g -O2 -MT fts-backend-xapian.lo -MD -MP -MF .deps/fts-backend-xapian.Tpo -c -o fts-backend-xapian.lo fts-backend-xapian.cpp libtool: compile: c++ -DHAVE_CONFIG_H -I. -I.. -I/opt/dovecot2.3/include/dovecot -I/usr/local/include -I/opt/dovecot2.3/include/dovecot -g -O2 -MT fts-backend-xapian.lo -MD -MP -MF .deps/fts-backend-xapian.Tpo -c fts-backend-xapian.cpp -fPIC -DPIC -o .libs/fts-backend-xapian.o fts-backend-xapian.cpp:486:14: warning: format string is not a string literal (potentially insecure) [-Wformat-security] i_warning(e.get_msg().c_str()); ^~~~~~~~~~~~~~~~~~~ fts-backend-xapian.cpp:486:14: note: treat the string as an argument to avoid this i_warning(e.get_msg().c_str()); ^ "%s", 1 warning generated.

Is that something you can look into as well?

On Mon, 14 Jan 2019 at 11:43, Joan Moreau jom@grosjo.net wrote:

THank you Odhiambo. I updated accordingly

On 2019-01-14 08:07, Odhiambo Washington wrote:

In your README.md, perhaps "This project intends to provide a straightforward and simple *procedure *to configure FTS plugin for Dovecot, leveraging the efforts by the Xapian.org team." is better?? Also in the part after cloning from git:

./configure --prefix=/usr --with-dovecot=/path/to/dovecot [ This /path/to/dovecot is not obvious. Is it the dovecot binary or what??]

On Mon, 14 Jan 2019 at 09:42, Joan Moreau via dovecot dovecot@dovecot.org wrote:

...

Thank you Stephan.

The version here shall be up and running : https://github.com/grosjo/fts-xapian

On 2019-01-14 00:07, Stephan Bosch wrote:

Op 13/01/2019 om 21:25 schreef Joan Moreau via dovecot:

I tried to combined it, the "autoreconf" errors are solved

Now, when I type "make install", the lib is not pushed into dovecot folder, but somewhere in /usr/local/...

How to adjust this to have it arriving in the proper folder ?

Depends on your system. It mostly a matter of setting a proper --prefix directory for configure, but other paths are configurable as well. I usually check what the official distribution package for Dovecot is doing and use that as a basis.

For Debian I use the following configure command:

./configure --with-ldap=plugin --with-ssl=openssl --with-sql=plugin --with-lua=plugin --with-pgsql --with-mysql --with-sqlite
--with-gssapi=plugin --with-solr --with-ioloop=best --enable-maintainer-mode
--prefix=/usr --sysconfdir=/etc --libexecdir=/usr/lib --localstatedir=/var --mandir=/usr/share/man
--infodir=/usr/share/info --with-moduledir=/usr/lib/dovecot/modules --disable-rpath --disable-static

Regards,

Stephan

On 2019-01-13 21:01, Tuomi, Aki wrote:

You copied your Makefile.am there. Stephan made you a working version, can you try that? (sorry for dup) Aki -------- Original message -------- From: Joan Moreau jom@grosjo.net Date: 13/01/2019 21:39 (GMT+02:00) To: Stephan Bosch stephan@rename-it.nl Cc: Aki Tuomi aki.tuomi@open-xchange.com Subject: Re: [FTS Xapian] Beta release

I used the skeleton from Aki : https://github.com/grosjo/fts-xapian

However, when I try to act as a visitor, I reach teh follwoing error:

# autoreconf -vi autoreconf: Entering directory `.' autoreconf: configure.ac: not using Gettext autoreconf: running: aclocal -I m4 autoreconf: configure.ac: tracing autoreconf: running: libtoolize --copy libtoolize: putting auxiliary files in AC_CONFIG_AUX_DIR, '.'. libtoolize: copying file './ltmain.sh' libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'm4'. libtoolize: copying file 'm4/libtool.m4' libtoolize: copying file 'm4/ltoptions.m4' libtoolize: copying file 'm4/ltsugar.m4' libtoolize: copying file 'm4/ltversion.m4' libtoolize: copying file 'm4/lt~obsolete.m4' autoreconf: running: /usr/bin/autoconf autoreconf: running: /usr/bin/autoheader autoreconf: running: automake --add-missing --copy --no-force configure.ac:9: installing './compile' configure.ac:11: installing './config.guess' configure.ac:11: installing './config.sub' configure.ac:7: installing './install-sh' configure.ac:7: installing './missing' src/Makefile.am: installing './depcomp' /usr/share/automake-1.16/am/depend2.am: error: am__fastdepCXX does not appear in AM_CONDITIONAL /usr/share/automake-1.16/am/depend2.am: The usual way to define 'am__fastdepCXX' is to add 'AC_PROG_CXX' /usr/share/automake-1.16/am/depend2.am: to 'configure.ac' and run 'aclocal' and 'autoconf' again src/Makefile.am: error: C++ source seen but 'CXX' is undefined src/Makefile.am: The usual way to define 'CXX' is to add 'AC_PROG_CXX' src/Makefile.am: to 'configure.ac' and run 'autoconf' again. src/Makefile.am:11: warning: variable 'NOPLUGIN_LDFLAGS' is defined but no program or src/Makefile.am:11: library has 'NOPLUGIN' as canonical name (possible typo) autoreconf: automake failed with exit status: 1

On 2019-01-13 20:24, Stephan Bosch wrote:

Oh, right, a distribution tarball doesn't include some of the
necessary files for your repository like autogen.sh and
.gitignore. The attached tarball includes all those and is ready
for `git init`. The previous tarball was made with `make
distcheck` from this one.

Regards,

Stephan.

Op 13/01/2019 om 20:14 schreef Stephan Bosch:

    Hi Joan,

    Op 13/01/2019 om 19:03 schreef Aki Tuomi:

        Yes, from compiling point of view it is done.

        Unfortunately what is not done is all the other work
        involved, such as fixing all the inevitable bugs it has
        and maintaining it. We do not want, at this moment, take
        up maintaining and developing yet another FTS plugin as
        we have plenty of things to do already.

        I invite you to setup your own repository and provide
        this plugin from there, being the maintainer of this
        plugin. We can add a link to your plugin on our FTS page
        so people can also find it.

        There are other plugins like this, e.g.
        https://github.com/st3fan/dovecot-xaps-plugin


    I turned the code you provided into a separate plugin
    package. The distribution tarball is attached.

    Notable changes:

    - Added example copyright headers and COPYING and AUTHORS
    files. You should modify those to your preference.
    - Added README and INSTALL files (in markdown using Pandoc).
    Those need to be amended with details.
    - Amended the plugin code to display a debug message with the
    plugin name and version upon plugin load.


    I advise you to turn this into a git repository and continue from there.

    I do not recommend releasing this plugin with the
    -fpermissive flag and the resulting warning as it is now. But
    I'm assuming this is still a work in progress, so that is OK.

    Regards,

    Stephan.


            On 13 January 2019 at 19:52 Joan Moreau <jom@grosjo.net
            <mailto:jom@grosjo.net>> wrote:


            The only point here of this fts-xapian is to get rid
            of solr (because it

            is just a nightmare to setup) and squat (because it is considere
            obsolete).

            I already sent the changed in configure.ac, makefile.am

, etc.. in order to include it in the dovecot, and it compiles properly

            The only remaining point is to push it in hte git (yes, everything is
            already done)

            On 2019-01-13 18:45, Aki Tuomi wrote:

                    On 13 January 2019 at 17:05 Joan Moreau via
                    dovecot <dovecot@dovecot.org
                    <mailto:dovecot@dovecot.org>> wrote:

                    Hi

                    Please find attached the beta release of FTS
                    Xapian, with the objective
                    to replace fts_squat that is being deprecated.


                    Configuration is exactly the same as for fts_squat:

                    plugin {

                    plugin = fts fts_xapian (...)
                    fts = xapian
                    fts_autoindex = yes
                    fts_enforced = yes
                    fts_xapian = partial=2 full=20

                    This is installed on my production server
                    (>120Gb of mailboxes), and I
                    will observe it during the coming days.


                    I will definitely appreciate that this is added in the core git of
                    docevot, in order to have a versionning of
                    it, to remove squat and let

                    basic users able to avoid Solr alternative as much as possible.

                    Thanks

                    JM

                Hi!

                I still recommend you setup a, say, github
                repository for your plugin. We are not able to
                currently include your work in dovecot core as it
                is more work than just pushing the code into the
                repo. Maybe it can be included in the future.

                If you want, I can help you in setting up the
                required configuration scripts and such to make
                it possible to compile it as plugin.

                Then anyone can download it and install it for
                their dovecot, even if dovecot itself has been
                installed from packages, and also makes it
                possible for package maintainers to consider
                including it in distributions.

                Aki

-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 "Oh, the cruft.", grep ^[^#] :-)

-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 "Oh, the cruft.", grep ^[^#] :-)

Op 14-1-2019 om 13:40 schreef Aki Tuomi:

Just to remind that now that there is a github repo for fts-xapian, you could maybe open these issues there instead?

Although README.md currently says:

"Please feel free to send your questions, together with the dovecot log file, to jom@grosjo.net mailto:jom@grosjo.net or to the dovecot ML dovecot@dovecot.org mailto:dovecot@dovecot.org."

Regards,

Stephan.

Aki

On 14.1.2019 14.29, Odhiambo Washington wrote:

...

Testing a compile on FreeBSD.

gmake[2]: Entering directory '/usr/home/wash/Tools/Dovecot/fts-xapian/src' /bin/sh ../libtool  --tag=CXX   --mode=compile c++ -DHAVE_CONFIG_H -I. -I.. -I/opt/dovecot2.3/include/dovecot -I/opt/dovecot2.3/include/dovecot      -g -O2 -MT fts-backend-xapian.lo -MD -MP -MF .deps/fts-backend-xapian.Tpo -c -o fts-backend-xapian.lo fts-backend-xapian.cpp libtool: compile:  c++ -DHAVE_CONFIG_H -I. -I.. -I/opt/dovecot2.3/include/dovecot -I/opt/dovecot2.3/include/dovecot -g -O2 -MT fts-backend-xapian.lo -MD -MP -MF .deps/fts-backend-xapian.Tpo -c fts-backend-xapian.cpp  -fPIC -DPIC -o .libs/fts-backend-xapian.o fts-backend-xapian.cpp:3:10: fatal error: 'xapian.h' file not found #include ^~~~~~~~~~

Well, I installed xapian-core and the xapian.h is in /usr/local/include/

I can overcome the fatal error by doing:

 env CPPFLAGS=-I/usr/local/include PANDOC=false ./configure --prefix=/opt --with-dovecot=/opt/dovecot2.3/lib/dovecot/

Is that something that you can address within the code or we (*BSD) have to live with it?

During make, the following warning is generated:

/bin/sh ../libtool  --tag=CXX   --mode=compile c++ -DHAVE_CONFIG_H -I. -I.. -I/opt/dovecot2.3/include/dovecot  -I/usr/local/include -I/opt/dovecot2.3/include/dovecot      -g -O2 -MT fts-backend-xapian.lo -MD -MP -MF .deps/fts-backend-xapian.Tpo -c -o fts-backend-xapian.lo fts-backend-xapian.cpp libtool: compile:  c++ -DHAVE_CONFIG_H -I. -I.. -I/opt/dovecot2.3/include/dovecot -I/usr/local/include -I/opt/dovecot2.3/include/dovecot -g -O2 -MT fts-backend-xapian.lo -MD -MP -MF .deps/fts-backend-xapian.Tpo -c fts-backend-xapian.cpp  -fPIC -DPIC -o .libs/fts-backend-xapian.o fts-backend-xapian.cpp:486:14: warning: format string is not a string literal (potentially insecure) [-Wformat-security] i_warning(e.get_msg().c_str()); ^~~~~~~~~~~~~~~~~~~ fts-backend-xapian.cpp:486:14: note: treat the string as an argument to avoid this i_warning(e.get_msg().c_str()); ^ "%s", 1 warning generated.

Is that something you can look into as well?

On Mon, 14 Jan 2019 at 11:43, Joan Moreau mailto:jom@grosjo.net> wrote:

THank you Odhiambo. I updated accordingly



On 2019-01-14 08:07, Odhiambo Washington wrote:

...

In your README.md, perhaps "This project intends to provide a
straightforward and simple *procedure *to configure FTS plugin
for Dovecot, leveraging the efforts by the Xapian.org team." is
better??
Also in the part after cloning from git:
./configure --prefix=/usr --with-dovecot=/path/to/dovecot [ This
/path/to/dovecot is not obvious. Is it the dovecot binary or what??]

On Mon, 14 Jan 2019 at 09:42, Joan Moreau via dovecot
<dovecot@dovecot.org <mailto:dovecot@dovecot.org>> wrote:

    Thank you Stephan.

    The version here shall be up and running :
    https://github.com/grosjo/fts-xapian



    On 2019-01-14 00:07, Stephan Bosch wrote:



        Op 13/01/2019 om 21:25 schreef Joan Moreau via dovecot:


            I tried to combined it, the "autoreconf" errors are solved

            Now, when I type "make install", the lib is not
            pushed into dovecot folder, but somewhere in
            /usr/local/...

            How to adjust this to have it arriving in the proper folder ?


        Depends on your system. It mostly a matter of setting a
        proper --prefix directory for configure, but other paths
        are configurable as well. I usually check what the
        official distribution package for Dovecot is doing and
        use that as a basis.

        For Debian I use the following configure command:

        ./configure --with-ldap=plugin --with-ssl=openssl
        --with-sql=plugin --with-lua=plugin --with-pgsql
        --with-mysql --with-sqlite \
                --with-gssapi=plugin --with-solr
        --with-ioloop=best --enable-maintainer-mode \
                --prefix=/usr --sysconfdir=/etc
        --libexecdir=/usr/lib --localstatedir=/var
        --mandir=/usr/share/man \
                --infodir=/usr/share/info
        --with-moduledir=/usr/lib/dovecot/modules
        --disable-rpath --disable-static

        Regards,

        Stephan


            On 2019-01-13 21:01, Tuomi, Aki wrote:

                You copied your Makefile.am there. Stephan made
                you a working version, can you try that?
                (sorry for dup)
                Aki
                -------- Original message --------
                From: Joan Moreau <jom@grosjo.net
                <mailto:jom@grosjo.net>>
                Date: 13/01/2019 21:39 (GMT+02:00)
                To: Stephan Bosch <stephan@rename-it.nl
                <mailto:stephan@rename-it.nl>>
                Cc: Aki Tuomi <aki.tuomi@open-xchange.com
                <mailto:aki.tuomi@open-xchange.com>>
                Subject: Re: [FTS Xapian] Beta release

                I used the skeleton from Aki :
                https://github.com/grosjo/fts-xapian

                However, when I try to act as a visitor, I reach teh follwoing error:

                # autoreconf -vi
                autoreconf: Entering directory `.'
                autoreconf: configure.ac
                <http://configure.ac>: not using Gettext
                autoreconf: running: aclocal -I m4
                autoreconf: configure.ac
                <http://configure.ac>: tracing
                autoreconf: running: libtoolize --copy
                libtoolize: putting auxiliary files in AC_CONFIG_AUX_DIR, '.'.
                libtoolize: copying file './ltmain.sh'
                libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'm4'.
                libtoolize: copying file 'm4/libtool.m4'
                libtoolize: copying file 'm4/ltoptions.m4'
                libtoolize: copying file 'm4/ltsugar.m4'
                libtoolize: copying file 'm4/ltversion.m4'
                libtoolize: copying file 'm4/lt~obsolete.m4'
                autoreconf: running: /usr/bin/autoconf
                autoreconf: running: /usr/bin/autoheader
                autoreconf: running: automake --add-missing --copy --no-force
                configure.ac:9
                <http://configure.ac:9>: installing './compile'
                configure.ac:11
                <http://configure.ac:11>: installing './config.guess'
                configure.ac:11
                <http://configure.ac:11>: installing './config.sub'
                configure.ac:7
                <http://configure.ac:7>: installing './install-sh'
                configure.ac:7
                <http://configure.ac:7>: installing './missing'
                src/Makefile.am: installing './depcomp'
                /usr/share/automake-1.16/am/depend2.am
                <http://depend2.am>: error: am__fastdepCXX does
                not appear in AM_CONDITIONAL
                /usr/share/automake-1.16/am/depend2.am
                <http://depend2.am>: The usual way to define
                'am__fastdepCXX' is to add 'AC_PROG_CXX'
                /usr/share/automake-1.16/am/depend2.am
                <http://depend2.am>: to 'configure.ac
                <http://configure.ac>' and run 'aclocal' and
                'autoconf' again
                src/Makefile.am: error: C++ source seen but 'CXX' is undefined
                src/Makefile.am: The usual way to define 'CXX' is to add 'AC_PROG_CXX'
                src/Makefile.am: to 'configure.ac
                <http://configure.ac>' and run 'autoconf' again.
                src/Makefile.am:11: warning: variable
                'NOPLUGIN_LDFLAGS' is defined but no program or
                src/Makefile.am:11: library has 'NOPLUGIN' as
                canonical name (possible typo)
                autoreconf: automake failed with exit status: 1



                On 2019-01-13 20:24, Stephan Bosch wrote:

                    Oh, right, a distribution tarball doesn't include some of the
                    necessary files for your repository like autogen.sh and
                    .gitignore. The attached tarball includes all those and is ready
                    for `git init`. The previous tarball was made with `make
                    distcheck` from this one.

                    Regards,

                    Stephan.

                    Op 13/01/2019 om 20:14 schreef Stephan Bosch:

                        Hi Joan,

                        Op 13/01/2019 om 19:03 schreef Aki Tuomi:

                            Yes, from compiling point of view it is done.

                            Unfortunately what is not done is all the other work
                            involved, such as fixing all the inevitable bugs it has
                            and maintaining it. We do not want, at this moment, take
                            up maintaining and developing yet another FTS plugin as
                            we have plenty of things to do already.

                            I invite you to setup your own repository and provide
                            this plugin from there, being the maintainer of this
                            plugin. We can add a link to your plugin on our FTS page
                            so people can also find it.

                            There are other plugins like this, e.g.
                https://github.com/st3fan/dovecot-xaps-plugin


                        I turned the code you provided into a separate plugin
                        package. The distribution tarball is attached.

                        Notable changes:

                        - Added example copyright headers and COPYING and AUTHORS
                        files. You should modify those to your preference.
                        - Added README and INSTALL files (in markdown using Pandoc).
                        Those need to be amended with details.
                        - Amended the plugin code to display a debug message with the
                        plugin name and version upon plugin load.

                        I advise you to turn this into a git repository and continue from there.

                        I do not recommend releasing this plugin with the
                        -fpermissive flag and the resulting warning as it is now. But
                        I'm assuming this is still a work in progress, so that is OK.

                        Regards,

                        Stephan.


                                On 13 January 2019 at 19:52 Joan Moreau <jom@grosjo.net
                <mailto:jom@grosjo.net>
                                <mailto:jom@grosjo.net
                <mailto:jom@grosjo.net>>> wrote:


                                The only point here of this fts-xapian is to get rid
                                of solr (because it
                                is just a nightmare to setup) and squat (because it is considere
                                obsolete).

                                I already sent the changed in
                configure.ac <http://configure.ac>, makefile.am
                <http://makefile.am>, etc.. in order
                                to include it in the dovecot, and it compiles properly

                                The only remaining point is to push it in hte git (yes, everything is
                                already done)

                                On 2019-01-13 18:45, Aki Tuomi wrote:

                                        On 13 January 2019 at 17:05 Joan Moreau via
                                        dovecot <dovecot@dovecot.org
                <mailto:dovecot@dovecot.org>
                                        <mailto:dovecot@dovecot.org
                <mailto:dovecot@dovecot.org>>> wrote:

                                        Hi

                                        Please find attached the beta release of FTS
                                        Xapian, with the objective
                                        to replace fts_squat that is being deprecated.

                                        Configuration is exactly the same as for fts_squat:

                                        plugin {

                                        plugin = fts fts_xapian (...)
                                        fts = xapian
                                        fts_autoindex = yes
                                        fts_enforced = yes
                                        fts_xapian = partial=2 full=20

                                        This is installed on my production server
                                        (>120Gb of mailboxes), and I
                                        will observe it during the coming days.

                                        I will definitely appreciate that this is added in the core git of
                                        docevot, in order to have a versionning of
                                        it, to remove squat and let
                            ��           basic users able to avoid Solr alternative as much as possible.

                                        Thanks

                                        JM

                                    Hi!

                                    I still recommend you setup a, say, github
                                    repository for your plugin. We are not able to
                                    currently include your work in dovecot core as it
                                    is more work than just pushing the code into the
                                    repo. Maybe it can be included in the future.

                                    If you want, I can help you in setting up the
                                    required configuration scripts and such to make
                                    it possible to compile it as plugin.

                                    Then anyone can download it and install it for
                                    their dovecot, even if dovecot itself has been
                                    installed from packages, and also makes it
                                    possible for package maintainers to consider
                                    including it in distributions.

                                    Aki




-- 
Best regards,
Odhiambo WASHINGTON,
Nairobi,KE
+254 7 3200 0004/+254 7 2274 3223
"Oh, the cruft.", grep ^[^#] :-)

-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 "Oh, the cruft.", grep ^[^#] :-)

It is indeed better is you use the issue tracker of github: https://github.com/grosjo/fts-xapian/issues

I updated the Readme accordingly

On 2019-01-14 14:24, Stephan Bosch wrote:

Op 14-1-2019 om 13:40 schreef Aki Tuomi:

...

Just to remind that now that there is a github repo for fts-xapian, you could maybe open these issues there instead? Although README.md currently says:

"Please feel free to send your questions, together with the dovecot log file, to jom@grosjo.net mailto:jom@grosjo.net or to the dovecot ML dovecot@dovecot.org mailto:dovecot@dovecot.org."

Regards,

Stephan.

Hi,

I installed and tested your version, but the indexer process crashes reproducible with the following command after about 2000 messages were indexed:

doveadm index -u paul@iwascoding.com -q \*

Jan 14 09:26:15 mail dovecot: indexer-worker(paul@iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Mailbox sent: UID 2038: Opened mail because: fts indexing Jan 14 09:26:15 mail dovecot: indexer-worker: Error: terminate called after throwing an instance of 'std::bad_alloc' Jan 14 09:26:15 mail dovecot: indexer-worker: Error: what(): std::bad_alloc Jan 14 09:26:15 mail dovecot: indexer: Error: Indexer worker disconnected, discarding 48 requests for paul@iwascoding.com Jan 14 09:26:15 mail dovecot: indexer-worker(paul@iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Fatal: master: service(indexer-worker): child 16777 killed with signal 6 (core dumps disabled - https://dovecot.org/bugreport.html#coredumps)

I tried to delete the message, but this does not help (crashes e.g. after message 2029 or 2044). Other folders with fewer messages were successfully indexed before.

Sorry, could not convince dovecot to create core dumps (read the docs, changed /proc/sys/kernel/core_pattern, added LimitCORE=unlimited/infinity, even created /etc/systemd/system/dovecot.service.d/coredump.conf to no avail). Custom Dovecot 2.3.4 on Debian Stretch.

Thanks, Paul

On 14. Jan 2019, at 07:42, Joan Moreau via dovecot dovecot@dovecot.org wrote:

Thank you Stephan.

The version here shall be up and running : https://github.com/grosjo/fts-xapian

On 2019-01-14 00:07, Stephan Bosch wrote:

...

Op 13/01/2019 om 21:25 schreef Joan Moreau via dovecot:

...

I tried to combined it, the "autoreconf" errors are solved

Now, when I type "make install", the lib is not pushed into dovecot folder, but somewhere in /usr/local/...

How to adjust this to have it arriving in the proper folder ?

Depends on your system. It mostly a matter of setting a proper --prefix directory for configure, but other paths are configurable as well. I usually check what the official distribution package for Dovecot is doing and use that as a basis.

For Debian I use the following configure command:

./configure --with-ldap=plugin --with-ssl=openssl --with-sql=plugin --with-lua=plugin --with-pgsql --with-mysql --with-sqlite
--with-gssapi=plugin --with-solr --with-ioloop=best --enable-maintainer-mode
--prefix=/usr --sysconfdir=/etc --libexecdir=/usr/lib --localstatedir=/var --mandir=/usr/share/man
--infodir=/usr/share/info --with-moduledir=/usr/lib/dovecot/modules --disable-rpath --disable-static

Regards,

Stephan

...

On 2019-01-13 21:01, Tuomi, Aki wrote:

...

You copied your Makefile.am there. Stephan made you a working version, can you try that? (sorry for dup) Aki -------- Original message -------- From: Joan Moreau jom@grosjo.net Date: 13/01/2019 21:39 (GMT+02:00) To: Stephan Bosch stephan@rename-it.nl Cc: Aki Tuomi aki.tuomi@open-xchange.com Subject: Re: [FTS Xapian] Beta release

I used the skeleton from Aki : https://github.com/grosjo/fts-xapian

However, when I try to act as a visitor, I reach teh follwoing error:

# autoreconf -vi autoreconf: Entering directory `.' autoreconf: configure.ac: not using Gettext autoreconf: running: aclocal -I m4 autoreconf: configure.ac: tracing autoreconf: running: libtoolize --copy libtoolize: putting auxiliary files in AC_CONFIG_AUX_DIR, '.'. libtoolize: copying file './ltmain.sh' libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'm4'. libtoolize: copying file 'm4/libtool.m4' libtoolize: copying file 'm4/ltoptions.m4' libtoolize: copying file 'm4/ltsugar.m4' libtoolize: copying file 'm4/ltversion.m4' libtoolize: copying file 'm4/lt~obsolete.m4' autoreconf: running: /usr/bin/autoconf autoreconf: running: /usr/bin/autoheader autoreconf: running: automake --add-missing --copy --no-force configure.ac:9: installing './compile' configure.ac:11: installing './config.guess' configure.ac:11: installing './config.sub' configure.ac:7: installing './install-sh' configure.ac:7: installing './missing' src/Makefile.am: installing './depcomp' /usr/share/automake-1.16/am/depend2.am: error: am__fastdepCXX does not appear in AM_CONDITIONAL /usr/share/automake-1.16/am/depend2.am: The usual way to define 'am__fastdepCXX' is to add 'AC_PROG_CXX' /usr/share/automake-1.16/am/depend2.am: to 'configure.ac' and run 'aclocal' and 'autoconf' again src/Makefile.am: error: C++ source seen but 'CXX' is undefined src/Makefile.am: The usual way to define 'CXX' is to add 'AC_PROG_CXX' src/Makefile.am: to 'configure.ac' and run 'autoconf' again. src/Makefile.am:11: warning: variable 'NOPLUGIN_LDFLAGS' is defined but no program or src/Makefile.am:11: library has 'NOPLUGIN' as canonical name (possible typo) autoreconf: automake failed with exit status: 1

On 2019-01-13 20:24, Stephan Bosch wrote:

Oh, right, a distribution tarball doesn't include some of the
necessary files for your repository like autogen.sh and
.gitignore. The attached tarball includes all those and is ready
for `git init`. The previous tarball was made with `make
distcheck` from this one.

Regards,

Stephan.

Op 13/01/2019 om 20:14 schreef Stephan Bosch:

    Hi Joan,

    Op 13/01/2019 om 19:03 schreef Aki Tuomi:

        Yes, from compiling point of view it is done.

        Unfortunately what is not done is all the other work
        involved, such as fixing all the inevitable bugs it has
        and maintaining it. We do not want, at this moment, take
        up maintaining and developing yet another FTS plugin as
        we have plenty of things to do already.

        I invite you to setup your own repository and provide
        this plugin from there, being the maintainer of this
        plugin. We can add a link to your plugin on our FTS page
        so people can also find it.

        There are other plugins like this, e.g.
        https://github.com/st3fan/dovecot-xaps-plugin


    I turned the code you provided into a separate plugin
    package. The distribution tarball is attached.

    Notable changes:

    - Added example copyright headers and COPYING and AUTHORS
    files. You should modify those to your preference.
    - Added README and INSTALL files (in markdown using Pandoc).
    Those need to be amended with details.
    - Amended the plugin code to display a debug message with the
    plugin name and version upon plugin load.

    I advise you to turn this into a git repository and continue from there.

    I do not recommend releasing this plugin with the
    -fpermissive flag and the resulting warning as it is now. But
    I'm assuming this is still a work in progress, so that is OK.

    Regards,

    Stephan.


            On 13 January 2019 at 19:52 Joan Moreau <jom@grosjo.net
            <mailto:jom@grosjo.net>> wrote:


            The only point here of this fts-xapian is to get rid
            of solr (because it
            is just a nightmare to setup) and squat (because it is considere
            obsolete).

            I already sent the changed in configure.ac, makefile.am, etc.. in order
            to include it in the dovecot, and it compiles properly

            The only remaining point is to push it in hte git (yes, everything is
            already done)

            On 2019-01-13 18:45, Aki Tuomi wrote:

                    On 13 January 2019 at 17:05 Joan Moreau via
                    dovecot <dovecot@dovecot.org
                    <mailto:dovecot@dovecot.org>> wrote:

                    Hi

                    Please find attached the beta release of FTS
                    Xapian, with the objective
                    to replace fts_squat that is being deprecated.

                    Configuration is exactly the same as for fts_squat:

                    plugin {

                    plugin = fts fts_xapian (...)
                    fts = xapian
                    fts_autoindex = yes
                    fts_enforced = yes
                    fts_xapian = partial=2 full=20

                    This is installed on my production server
                    (>120Gb of mailboxes), and I
                    will observe it during the coming days.

                    I will definitely appreciate that this is added in the core git of
                    docevot, in order to have a versionning of
                    it, to remove squat and let
                    basic users able to avoid Solr alternative as much as possible.

                    Thanks

                    JM

                Hi!

                I still recommend you setup a, say, github
                repository for your plugin. We are not able to
                currently include your work in dovecot core as it
                is more work than just pushing the code into the
                repo. Maybe it can be included in the future.

                If you want, I can help you in setting up the
                required configuration scripts and such to make
                it possible to compile it as plugin.

                Then anyone can download it and install it for
                their dovecot, even if dovecot itself has been
                installed from packages, and also makes it
                possible for package maintainers to consider
                including it in distributions.

                Aki

Paul Hecker paul@iwascoding.com http://www.iwascoding.com

iwascoding GmbH Weserstr 18, 10247 Berlin HRB 97269 B Amtsgericht Berlin-Charlottenburg Geschäftsführer: Paul Hecker, Ilja Iwas

Here it is:

Jan 14 09:26:08 mail dovecot: indexer-worker(paul@iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Effective uid=8, gid=8, home=/var/spool/mail/iwascoding/paul Jan 14 09:26:08 mail dovecot: indexer-worker(paul@iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Quota root: name=User quota backend=dict args=:file:/var/spool/mail/iwascoding/paul/dovecot-quota Jan 14 09:26:08 mail dovecot: indexer-worker(paul@iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Quota rule: root=User quota mailbox=* bytes=2147483648 messages=0 Jan 14 09:26:08 mail dovecot: indexer-worker(paul@iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Quota rule: root=User quota mailbox=* bytes=2147483648 messages=60000 Jan 14 09:26:08 mail dovecot: indexer-worker(paul@iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Quota grace: root=User quota bytes=214748364 (10%) Jan 14 09:26:08 mail dovecot: indexer-worker(paul@iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: dict quota: user=paul@iwascoding.com, uri=file:/var/spool/mail/iwascoding/paul/dovecot-quota, noenforcing=0 Jan 14 09:26:08 mail dovecot: indexer-worker(paul@iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Namespace inbox: type=private, prefix=, sep=/, inbox=yes, hidden=no, list=yes, subscriptions=yes location=mdbox:~/mdbox Jan 14 09:26:08 mail dovecot: indexer-worker(paul@iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: fs: root=/var/spool/mail/iwascoding/paul/mdbox, index=, indexpvt=, control=, inbox=, alt= Jan 14 09:26:08 mail dovecot: indexer-worker(paul@iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: FTS Xapian: Partial=2, Full=20 DB_PATH=/var/spool/mail/iwascoding/paul/mdbox/xapian-indexes Jan 14 09:26:08 mail dovecot: indexer-worker(paul@iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: quota: quota_over_flag check: quota_over_script unset - skipping Jan 14 09:26:08 mail dovecot: indexer-worker(paul@iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Mailbox sent: Mailbox opened because: indexing Jan 14 09:26:08 mail dovecot: indexer-worker(paul@iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: FTS Xapian : Mailbox sent : Last UID=0 Jan 14 09:26:08 mail dovecot: indexer-worker(paul@iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: FTS Xapian : Mailbox sent : Last UID=0 Jan 14 09:26:08 mail dovecot: indexer-worker(paul@iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Namespace : Using permissions from /var/spool/mail/iwascoding/paul/mdbox: mode=0700 gid=default Jan 14 09:26:08 mail dovecot: indexer-worker(paul@iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Mailbox sent: UID 1: Opened mail because: fts indexing Jan 14 09:26:08 mail dovecot: indexer-worker(paul@iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Opening RW /var/spool/mail/iwascoding/paul/mdbox/xapian-indexes/db_9ddfe10d8a8a8a568c120000654d370e Jan 14 09:26:08 mail dovecot: indexer-worker(paul@iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Mailbox sent: UID 2: Opened mail because: fts indexing Jan 14 09:26:08 mail dovecot: indexer-worker(paul@iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Mailbox sent: UID 3: Opened mail because: fts indexing

Thank you!

On 14. Jan 2019, at 10:11, Joan Moreau via dovecot dovecot@dovecot.org wrote:

Can you send the log part that includes the "init" of the plugins (something similar as below) ?

WHich version of Xapian are you on ?

Jan 14 09:10:04 gjserver dovecot[31082]: indexer-worker(admin@grosjo.net)<14725>apZfHexRPFy9QAAA0thIag:UL+TNOxRPFyFOQAA0thIag: FTS Xapian: Partial=2, Full=20 DB_PATH=/data/mail/grosjo.net/admin/xapian-indexes Jan 14 09:10:04 gjserver dovecot[31082]: indexer-worker(admin@grosjo.net)<14725>apZfHexRPFy9QAAA0thIag:UL+TNOxRPFyFOQAA0thIag: FTS Xapian : Mailbox Mail : Last UID=815055 Jan 14 09:10:04 gjserver dovecot[31082]: indexer-worker(admin@grosjo.net)<14725>apZfHexRPFy9QAAA0thIag:UL+TNOxRPFyFOQAA0thIag: FTS Xapian : Mailbox Mail : Last UID=815055 Jan 14 09:10:04 gjserver dovecot[31082]: indexer-worker(admin@grosjo.net)<14725>apZfHexRPFy9QAAA0thIag:UL+TNOxRPFyFOQAA0thIag: Opening RW /data/mail/grosjo.net/admin/xapian-indexes/db_5c935034609bc14c0e550000d6a3092d

On 2019-01-14 10:08, Paul Hecker via dovecot wrote:

...

Hi,

I installed and tested your version, but the indexer process crashes reproducible with the following command after about 2000 messages were indexed:

doveadm index -u paul@iwascoding.com mailto:paul@iwascoding.com -q \*

Jan 14 09:26:15 mail dovecot: indexer-worker(paul@iwascoding.com mailto:paul@iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Mailbox sent: UID 2038: Opened mail because: fts indexing Jan 14 09:26:15 mail dovecot: indexer-worker: Error: terminate called after throwing an instance of 'std::bad_alloc' Jan 14 09:26:15 mail dovecot: indexer-worker: Error: what(): std::bad_alloc Jan 14 09:26:15 mail dovecot: indexer: Error: Indexer worker disconnected, discarding 48 requests for paul@iwascoding.com mailto:paul@iwascoding.com Jan 14 09:26:15 mail dovecot: indexer-worker(paul@iwascoding.com mailto:paul@iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Fatal: master: service(indexer-worker): child 16777 killed with signal 6 (core dumps disabled - https://dovecot.org/bugreport.html#coredumps https://dovecot.org/bugreport.html#coredumps)

I tried to delete the message, but this does not help (crashes e.g. after message 2029 or 2044). Other folders with fewer messages were successfully indexed before.

Sorry, could not convince dovecot to create core dumps (read the docs, changed /proc/sys/kernel/core_pattern, added LimitCORE=unlimited/infinity, even created /etc/systemd/system/dovecot.service.d/coredump.conf to no avail). Custom Dovecot 2.3.4 on Debian Stretch.

Thanks, Paul

...

On 14. Jan 2019, at 07:42, Joan Moreau via dovecot mailto:dovecot@dovecot.org> wrote:

Thank you Stephan.

The version here shall be up and running : https://github.com/grosjo/fts-xapian https://github.com/grosjo/fts-xapian

On 2019-01-14 00:07, Stephan Bosch wrote:

...

Op 13/01/2019 om 21:25 schreef Joan Moreau via dovecot:

...

I tried to combined it, the "autoreconf" errors are solved

Now, when I type "make install", the lib is not pushed into dovecot folder, but somewhere in /usr/local/...

How to adjust this to have it arriving in the proper folder ?

Depends on your system. It mostly a matter of setting a proper --prefix directory for configure, but other paths are configurable as well. I usually check what the official distribution package for Dovecot is doing and use that as a basis.

For Debian I use the following configure command:

./configure --with-ldap=plugin --with-ssl=openssl --with-sql=plugin --with-lua=plugin --with-pgsql --with-mysql --with-sqlite
--with-gssapi=plugin --with-solr --with-ioloop=best --enable-maintainer-mode
--prefix=/usr --sysconfdir=/etc --libexecdir=/usr/lib --localstatedir=/var --mandir=/usr/share/man
--infodir=/usr/share/info --with-moduledir=/usr/lib/dovecot/modules --disable-rpath --disable-static

Regards,

Stephan

...

On 2019-01-13 21:01, Tuomi, Aki wrote:

...

You copied your Makefile.am there. Stephan made you a working version, can you try that? (sorry for dup) Aki -------- Original message -------- From: Joan Moreau mailto:jom@grosjo.net> Date: 13/01/2019 21:39 (GMT+02:00) To: Stephan Bosch mailto:stephan@rename-it.nl> Cc: Aki Tuomi mailto:aki.tuomi@open-xchange.com> Subject: Re: [FTS Xapian] Beta release

I used the skeleton from Aki : https://github.com/grosjo/fts-xapian https://github.com/grosjo/fts-xapian

However, when I try to act as a visitor, I reach teh follwoing error:

# autoreconf -vi autoreconf: Entering directory `.' autoreconf: configure.ac: not using Gettext autoreconf: running: aclocal -I m4 autoreconf: configure.ac: tracing autoreconf: running: libtoolize --copy libtoolize: putting auxiliary files in AC_CONFIG_AUX_DIR, '.'. libtoolize: copying file './ltmain.sh' libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'm4'. libtoolize: copying file 'm4/libtool.m4' libtoolize: copying file 'm4/ltoptions.m4' libtoolize: copying file 'm4/ltsugar.m4' libtoolize: copying file 'm4/ltversion.m4' libtoolize: copying file 'm4/lt~obsolete.m4' autoreconf: running: /usr/bin/autoconf autoreconf: running: /usr/bin/autoheader autoreconf: running: automake --add-missing --copy --no-force configure.ac:9: installing './compile' configure.ac:11: installing './config.guess' configure.ac:11: installing './config.sub' configure.ac:7: installing './install-sh' configure.ac:7: installing './missing' src/Makefile.am: installing './depcomp' /usr/share/automake-1.16/am/depend2.am: error: am__fastdepCXX does not appear in AM_CONDITIONAL /usr/share/automake-1.16/am/depend2.am: The usual way to define 'am__fastdepCXX' is to add 'AC_PROG_CXX' /usr/share/automake-1.16/am/depend2.am: to 'configure.ac' and run 'aclocal' and 'autoconf' again src/Makefile.am: error: C++ source seen but 'CXX' is undefined src/Makefile.am: The usual way to define 'CXX' is to add 'AC_PROG_CXX' src/Makefile.am: to 'configure.ac' and run 'autoconf' again. src/Makefile.am:11: warning: variable 'NOPLUGIN_LDFLAGS' is defined but no program or src/Makefile.am:11: library has 'NOPLUGIN' as canonical name (possible typo) autoreconf: automake failed with exit status: 1

On 2019-01-13 20:24, Stephan Bosch wrote:

Oh, right, a distribution tarball doesn't include some of the
necessary files for your repository like autogen.sh and
.gitignore. The attached tarball includes all those and is ready
for `git init`. The previous tarball was made with `make
distcheck` from this one.

Regards,

Stephan.

Op 13/01/2019 om 20:14 schreef Stephan Bosch:

    Hi Joan,

    Op 13/01/2019 om 19:03 schreef Aki Tuomi:

        Yes, from compiling point of view it is done.

        Unfortunately what is not done is all the other work
        involved, such as fixing all the inevitable bugs it has
        and maintaining it. We do not want, at this moment, take
        up maintaining and developing yet another FTS plugin as
        we have plenty of things to do already.

        I invite you to setup your own repository and provide
        this plugin from there, being the maintainer of this
        plugin. We can add a link to your plugin on our FTS page
        so people can also find it.

        There are other plugins like this, e.g.
        https://github.com/st3fan/dovecot-xaps-plugin <https://github.com/st3fan/dovecot-xaps-plugin>


    I turned the code you provided into a separate plugin
    package. The distribution tarball is attached.

    Notable changes:

    - Added example copyright headers and COPYING and AUTHORS
    files. You should modify those to your preference.
    - Added README and INSTALL files (in markdown using Pandoc).
    Those need to be amended with details.
    - Amended the plugin code to display a debug message with the
    plugin name and version upon plugin load.

    I advise you to turn this into a git repository and continue from there.

    I do not recommend releasing this plugin with the
    -fpermissive flag and the resulting warning as it is now. But
    I'm assuming this is still a work in progress, so that is OK.

    Regards,

    Stephan.


            On 13 January 2019 at 19:52 Joan Moreau <jom@grosjo.net <mailto:jom@grosjo.net>
            <mailto:jom@grosjo.net <mailto:jom@grosjo.net>>> wrote:


            The only point here of this fts-xapian is to get rid
            of solr (because it
            is just a nightmare to setup) and squat (because it is considere
            obsolete).

            I already sent the changed in configure.ac, makefile.am, etc.. in order
            to include it in the dovecot, and it compiles properly

            The only remaining point is to push it in hte git (yes, everything is
            already done)

            On 2019-01-13 18:45, Aki Tuomi wrote:

                    On 13 January 2019 at 17:05 Joan Moreau via
                    dovecot <dovecot@dovecot.org <mailto:dovecot@dovecot.org>
                    <mailto:dovecot@dovecot.org <mailto:dovecot@dovecot.org>>> wrote:

                    Hi

                    Please find attached the beta release of FTS
                    Xapian, with the objective
                    to replace fts_squat that is being deprecated.

                    Configuration is exactly the same as for fts_squat:

                    plugin {

                    plugin = fts fts_xapian (...)
                    fts = xapian
                    fts_autoindex = yes
                    fts_enforced = yes
                    fts_xapian = partial=2 full=20

                    This is installed on my production server
                    (>120Gb of mailboxes), and I
                    will observe it during the coming days.

                    I will definitely appreciate that this is added in the core git of
                    docevot, in order to have a versionning of
                    it, to remove squat and let
                    basic users able to avoid Solr alternative as much as possible.

                    Thanks

                    JM

                Hi!

                I still recommend you setup a, say, github
                repository for your plugin. We are not able to
                currently include your work in dovecot core as it
                is more work than just pushing the code into the
                repo. Maybe it can be included in the future.

                If you want, I can help you in setting up the
                required configuration scripts and such to make
                it possible to compile it as plugin.

                Then anyone can download it and install it for
                their dovecot, even if dovecot itself has been
                installed from packages, and also makes it
                possible for package maintainers to consider
                including it in distributions.

                Aki

Paul Hecker paul@iwascoding.com mailto:paul@iwascoding.com http://www.iwascoding.com http://www.iwascoding.com/

iwascoding GmbH Weserstr 18, 10247 Berlin HRB 97269 B Amtsgericht Berlin-Charlottenburg Geschäftsführer: Paul Hecker, Ilja Iwas

Paul Hecker paul@iwascoding.com mailto:paul@iwascoding.com http://www.iwascoding.com http://www.iwascoding.com/

iwascoding GmbH Weserstr 18, 10247 Berlin HRB 97269 B Amtsgericht Berlin-Charlottenburg Geschäftsführer: Paul Hecker, Ilja Iwas

Difficult to figure out without a coredump + gdb

I have also battled quite a lot to make sure dovecot can core dump on my Archlinux servers.

I remember that the key point was putting FS.SUID_DUMPABLE=2 in /etc/sysctl.d/ conf files, LIMITCORE=INFINITY in /etc/systemd/system/multi-user.target.wants/dovecot.service, and rebooting the server.

My own coredumps are on /var/lib/systemd/coredump/

On 2019-01-14 10:20, Paul Hecker via dovecot wrote:

Here it is:

Jan 14 09:26:08 mail dovecot: indexer-worker(paul@iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Effective uid=8, gid=8, home=/var/spool/mail/iwascoding/paul Jan 14 09:26:08 mail dovecot: indexer-worker(paul@iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Quota root: name=User quota backend=dict args=:file:/var/spool/mail/iwascoding/paul/dovecot-quota Jan 14 09:26:08 mail dovecot: indexer-worker(paul@iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Quota rule: root=User quota mailbox=* bytes=2147483648 messages=0 Jan 14 09:26:08 mail dovecot: indexer-worker(paul@iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Quota rule: root=User quota mailbox=* bytes=2147483648 messages=60000 Jan 14 09:26:08 mail dovecot: indexer-worker(paul@iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Quota grace: root=User quota bytes=214748364 (10%) Jan 14 09:26:08 mail dovecot: indexer-worker(paul@iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: dict quota: user=paul@iwascoding.com, uri=file:/var/spool/mail/iwascoding/paul/dovecot-quota, noenforcing=0 Jan 14 09:26:08 mail dovecot: indexer-worker(paul@iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Namespace inbox: type=private, prefix=, sep=/, inbox=yes, hidden=no, list=yes, subscriptions=yes location=mdbox:~/mdbox Jan 14 09:26:08 mail dovecot: indexer-worker(paul@iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: fs: root=/var/spool/mail/iwascoding/paul/mdbox, index=, indexpvt=, control=, inbox=, alt= Jan 14 09:26:08 mail dovecot: indexer-worker(paul@iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: FTS Xapian: Partial=2, Full=20 DB_PATH=/var/spool/mail/iwascoding/paul/mdbox/xapian-indexes Jan 14 09:26:08 mail dovecot: indexer-worker(paul@iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: quota: quota_over_flag check: quota_over_script unset - skipping Jan 14 09:26:08 mail dovecot: indexer-worker(paul@iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Mailbox sent: Mailbox opened because: indexing Jan 14 09:26:08 mail dovecot: indexer-worker(paul@iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: FTS Xapian : Mailbox sent : Last UID=0 Jan 14 09:26:08 mail dovecot: indexer-worker(paul@iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: FTS Xapian : Mailbox sent : Last UID=0 Jan 14 09:26:08 mail dovecot: indexer-worker(paul@iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Namespace : Using permissions from /var/spool/mail/iwascoding/paul/mdbox: mode=0700 gid=default Jan 14 09:26:08 mail dovecot: indexer-worker(paul@iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Mailbox sent: UID 1: Opened mail because: fts indexing Jan 14 09:26:08 mail dovecot: indexer-worker(paul@iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Opening RW /var/spool/mail/iwascoding/paul/mdbox/xapian-indexes/db_9ddfe10d8a8a8a568c120000654d370e Jan 14 09:26:08 mail dovecot: indexer-worker(paul@iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Mailbox sent: UID 2: Opened mail because: fts indexing Jan 14 09:26:08 mail dovecot: indexer-worker(paul@iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Mailbox sent: UID 3: Opened mail because: fts indexing

Thank you!

On 14. Jan 2019, at 10:11, Joan Moreau via dovecot dovecot@dovecot.org wrote:

Can you send the log part that includes the "init" of the plugins (something similar as below) ?

WHich version of Xapian are you on ?

Jan 14 09:10:04 gjserver dovecot[31082]: indexer-worker(admin@grosjo.net)<14725>apZfHexRPFy9QAAA0thIag:UL+TNOxRPFyFOQAA0thIag: FTS Xapian: Partial=2, Full=20 DB_PATH=/data/mail/grosjo.net/admin/xapian-indexes [1] Jan 14 09:10:04 gjserver dovecot[31082]: indexer-worker(admin@grosjo.net)<14725>apZfHexRPFy9QAAA0thIag:UL+TNOxRPFyFOQAA0thIag: FTS Xapian : Mailbox Mail : Last UID=815055 Jan 14 09:10:04 gjserver dovecot[31082]: indexer-worker(admin@grosjo.net)<14725>apZfHexRPFy9QAAA0thIag:UL+TNOxRPFyFOQAA0thIag: FTS Xapian : Mailbox Mail : Last UID=815055 Jan 14 09:10:04 gjserver dovecot[31082]: indexer-worker(admin@grosjo.net)<14725>apZfHexRPFy9QAAA0thIag:UL+TNOxRPFyFOQAA0thIag: Opening RW /data/mail/grosjo.net/admin/xapian-indexes/db_5c935034609bc14c0e550000d6a3092d [2]

On 2019-01-14 10:08, Paul Hecker via dovecot wrote: Hi,

I installed and tested your version, but the indexer process crashes reproducible with the following command after about 2000 messages were indexed:

doveadm index -u paul@iwascoding.com -q \*

Jan 14 09:26:15 mail dovecot: indexer-worker(paul@iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Mailbox sent: UID 2038: Opened mail because: fts indexing Jan 14 09:26:15 mail dovecot: indexer-worker: Error: terminate called after throwing an instance of 'std::bad_alloc' Jan 14 09:26:15 mail dovecot: indexer-worker: Error: what(): std::bad_alloc Jan 14 09:26:15 mail dovecot: indexer: Error: Indexer worker disconnected, discarding 48 requests for paul@iwascoding.com Jan 14 09:26:15 mail dovecot: indexer-worker(paul@iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Fatal: master: service(indexer-worker): child 16777 killed with signal 6 (core dumps disabled - https://dovecot.org/bugreport.html#coredumps)

I tried to delete the message, but this does not help (crashes e.g. after message 2029 or 2044). Other folders with fewer messages were successfully indexed before.

Sorry, could not convince dovecot to create core dumps (read the docs, changed /proc/sys/kernel/core_pattern, added LimitCORE=unlimited/infinity, even created /etc/systemd/system/dovecot.service.d/coredump.conf to no avail). Custom Dovecot 2.3.4 on Debian Stretch.

Thanks, Paul

Links:

[1] http://grosjo.net/admin/xapian-indexes [2] http://grosjo.net/admin/xapian-indexes/db_5c935034609bc14c0e550000d6a3092d

Op 14-1-2019 om 10:55 schreef Paul Hecker via dovecot:

OK, got it (my fault, as always, put the LimitCORE in the wrong line). Here is the stack trace:

If you want to get rid of those "??" stack trace elements, you'll need to install debug symbols for the xapian library. It depends on your system how to do that (usually some separate package).

Regards,

Stephan.

...

On 14. Jan 2019, at 10:33, Joan Moreau via dovecot mailto:dovecot@dovecot.org> wrote:

Difficult to figure out without a coredump + gdb

I have also battled quite a lot to make sure dovecot can core dump on my Archlinux servers.

I remember that the key point was putting*fs.suid_dumpable=2* in /etc/sysctl.d/ conf files, *LimitCORE=infinity* in /etc/systemd/system/multi-user.target.wants/dovecot.service,  and rebooting the server.

My own coredumps are on /var/lib/systemd/coredump/

THanks Paul

Can you try indexing the same emails with "full=10" for instance in the dovecot.conf ?

On 2019-01-14 12:19, Paul Hecker via dovecot wrote:

Thank you. Here is the stack trace with all debug symbols:

On 14. Jan 2019, at 11:07, Stephan Bosch stephan@rename-it.nl wrote:

Op 14-1-2019 om 10:55 schreef Paul Hecker via dovecot: OK, got it (my fault, as always, put the LimitCORE in the wrong line). Here is the stack trace:

If you want to get rid of those "??" stack trace elements, you'll need to install debug symbols for the xapian library. It depends on your system how to do that (usually some separate package).

Regards,

Stephan.

On 14. Jan 2019, at 10:33, Joan Moreau via dovecot mailto:dovecot@dovecot.org> wrote:

Difficult to figure out without a coredump + gdb

I have also battled quite a lot to make sure dovecot can core dump on my Archlinux servers.

I remember that the key point was putting*fs.suid_dumpable=2* in /etc/sysctl.d/ conf files, *LimitCORE=infinity* in /etc/systemd/system/multi-user.target.wants/dovecot.service, and rebooting the server.

My own coredumps are on /var/lib/systemd/coredump/