[FTS Xapian] Beta release

Mon Jan 14 11:11:39 EET 2019

Can you send the log part that includes the "init" of the plugins
(something similar as below) ? 

WHich version of Xapian are you on ? 

Jan 14 09:10:04 gjserver dovecot[31082]:
indexer-worker(admin at grosjo.net)<14725><apZfHexRPFy9QAAA0thIag:UL+TNOxRPFyFOQAA0thIag>:
FTS Xapian: Partial=2, Full=20
DB_PATH=/data/mail/grosjo.net/admin/xapian-indexes
Jan 14 09:10:04 gjserver dovecot[31082]:
indexer-worker(admin at grosjo.net)<14725><apZfHexRPFy9QAAA0thIag:UL+TNOxRPFyFOQAA0thIag>:
FTS Xapian : Mailbox Mail : Last UID=815055
Jan 14 09:10:04 gjserver dovecot[31082]:
indexer-worker(admin at grosjo.net)<14725><apZfHexRPFy9QAAA0thIag:UL+TNOxRPFyFOQAA0thIag>:
FTS Xapian : Mailbox Mail : Last UID=815055
Jan 14 09:10:04 gjserver dovecot[31082]:
indexer-worker(admin at grosjo.net)<14725><apZfHexRPFy9QAAA0thIag:UL+TNOxRPFyFOQAA0thIag>:
Opening RW
/data/mail/grosjo.net/admin/xapian-indexes/db_5c935034609bc14c0e550000d6a3092d

On 2019-01-14 10:08, Paul Hecker via dovecot wrote:

> Hi,
> 
> I installed and tested your version, but the indexer process crashes reproducible with the following command after about 2000 messages were indexed:
> 
> doveadm index -u paul at iwascoding.com -q \*
> 
> Jan 14 09:26:15 mail dovecot: indexer-worker(paul at iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Mailbox sent: UID 2038: Opened mail because: fts indexing
> Jan 14 09:26:15 mail dovecot: indexer-worker: Error: terminate called after throwing an instance of 'std::bad_alloc'
> Jan 14 09:26:15 mail dovecot: indexer-worker: Error:   what():  std::bad_alloc
> Jan 14 09:26:15 mail dovecot: indexer: Error: Indexer worker disconnected, discarding 48 requests for paul at iwascoding.com
> Jan 14 09:26:15 mail dovecot: indexer-worker(paul at iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Fatal: master: service(indexer-worker): child 16777 killed with signal 6 (core dumps disabled - https://dovecot.org/bugreport.html#coredumps)
> 
> I tried to delete the message, but this does not help (crashes e.g. after message 2029 or 2044). Other folders with fewer messages were successfully indexed before.
> 
> Sorry, could not convince dovecot to create core dumps (read the docs, changed /proc/sys/kernel/core_pattern, added LimitCORE=unlimited/infinity, even created /etc/systemd/system/dovecot.service.d/coredump.conf to no avail). Custom Dovecot 2.3.4 on Debian Stretch.
> 
> Thanks,
> Paul
> 
> On 14. Jan 2019, at 07:42, Joan Moreau via dovecot <dovecot at dovecot.org> wrote:
> 
> Thank you Stephan.
> 
> The version here shall be up and running : https://github.com/grosjo/fts-xapian
> 
> On 2019-01-14 00:07, Stephan Bosch wrote:
> 
> Op 13/01/2019 om 21:25 schreef Joan Moreau via dovecot: 
> I tried to combined it, the "autoreconf" errors are solved
> 
> Now, when I type "make install", the lib is not pushed into dovecot folder, but somewhere in /usr/local/...
> 
> How to adjust this to have it arriving in the proper folder ?
> 
> Depends on your system. It mostly a matter of setting a proper --prefix directory for configure, but other paths are configurable as well. I usually check what the official distribution package for Dovecot is doing and use that as a basis.
> 
> For Debian I use the following configure command:
> 
> ./configure --with-ldap=plugin --with-ssl=openssl --with-sql=plugin --with-lua=plugin --with-pgsql --with-mysql --with-sqlite \
> --with-gssapi=plugin --with-solr --with-ioloop=best --enable-maintainer-mode \
> --prefix=/usr --sysconfdir=/etc --libexecdir=/usr/lib --localstatedir=/var --mandir=/usr/share/man \
> --infodir=/usr/share/info --with-moduledir=/usr/lib/dovecot/modules --disable-rpath --disable-static
> 
> Regards,
> 
> Stephan
> 
> On 2019-01-13 21:01, Tuomi, Aki wrote:
> 
> You copied your Makefile.am there. Stephan made you a working version, can you try that?
> (sorry for dup)
> Aki
> -------- Original message --------
> From: Joan Moreau <jom at grosjo.net>
> Date: 13/01/2019 21:39 (GMT+02:00)
> To: Stephan Bosch <stephan at rename-it.nl>
> Cc: Aki Tuomi <aki.tuomi at open-xchange.com>
> Subject: Re: [FTS Xapian] Beta release
> 
> I used the skeleton from Aki : https://github.com/grosjo/fts-xapian
> 
> However, when I try to act as a visitor, I reach teh follwoing error:
> 
> # autoreconf -vi
> autoreconf: Entering directory `.'
> autoreconf: configure.ac: not using Gettext
> autoreconf: running: aclocal -I m4
> autoreconf: configure.ac: tracing
> autoreconf: running: libtoolize --copy
> libtoolize: putting auxiliary files in AC_CONFIG_AUX_DIR, '.'.
> libtoolize: copying file './ltmain.sh'
> libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'm4'.
> libtoolize: copying file 'm4/libtool.m4'
> libtoolize: copying file 'm4/ltoptions.m4'
> libtoolize: copying file 'm4/ltsugar.m4'
> libtoolize: copying file 'm4/ltversion.m4'
> libtoolize: copying file 'm4/lt~obsolete.m4'
> autoreconf: running: /usr/bin/autoconf
> autoreconf: running: /usr/bin/autoheader
> autoreconf: running: automake --add-missing --copy --no-force
> configure.ac:9: installing './compile'
> configure.ac:11: installing './config.guess'
> configure.ac:11: installing './config.sub'
> configure.ac:7: installing './install-sh'
> configure.ac:7: installing './missing'
> src/Makefile.am: installing './depcomp'
> /usr/share/automake-1.16/am/depend2.am: error: am__fastdepCXX does not appear in AM_CONDITIONAL
> /usr/share/automake-1.16/am/depend2.am: The usual way to define 'am__fastdepCXX' is to add 'AC_PROG_CXX'
> /usr/share/automake-1.16/am/depend2.am: to 'configure.ac' and run 'aclocal' and 'autoconf' again
> src/Makefile.am: error: C++ source seen but 'CXX' is undefined
> src/Makefile.am: The usual way to define 'CXX' is to add 'AC_PROG_CXX'
> src/Makefile.am: to 'configure.ac' and run 'autoconf' again.
> src/Makefile.am:11: warning: variable 'NOPLUGIN_LDFLAGS' is defined but no program or
> src/Makefile.am:11: library has 'NOPLUGIN' as canonical name (possible typo)
> autoreconf: automake failed with exit status: 1
> 
> On 2019-01-13 20:24, Stephan Bosch wrote:
> 
> Oh, right, a distribution tarball doesn't include some of the
> necessary files for your repository like autogen.sh and
> .gitignore. The attached tarball includes all those and is ready
> for `git init`. The previous tarball was made with `make
> distcheck` from this one.
> 
> Regards,
> 
> Stephan.
> 
> Op 13/01/2019 om 20:14 schreef Stephan Bosch:
> 
> Hi Joan,
> 
> Op 13/01/2019 om 19:03 schreef Aki Tuomi:
> 
> Yes, from compiling point of view it is done.
> 
> Unfortunately what is not done is all the other work
> involved, such as fixing all the inevitable bugs it has
> and maintaining it. We do not want, at this moment, take
> up maintaining and developing yet another FTS plugin as
> we have plenty of things to do already.
> 
> I invite you to setup your own repository and provide
> this plugin from there, being the maintainer of this
> plugin. We can add a link to your plugin on our FTS page
> so people can also find it.
> 
> There are other plugins like this, e.g.
> https://github.com/st3fan/dovecot-xaps-plugin
> 
> I turned the code you provided into a separate plugin
> package. The distribution tarball is attached.
> 
> Notable changes:
> 
> - Added example copyright headers and COPYING and AUTHORS
> files. You should modify those to your preference.
> - Added README and INSTALL files (in markdown using Pandoc).
> Those need to be amended with details.
> - Amended the plugin code to display a debug message with the
> plugin name and version upon plugin load.
> 
> I advise you to turn this into a git repository and continue from there.
> 
> I do not recommend releasing this plugin with the
> -fpermissive flag and the resulting warning as it is now. But
> I'm assuming this is still a work in progress, so that is OK.
> 
> Regards,
> 
> Stephan.
> 
> On 13 January 2019 at 19:52 Joan Moreau <jom at grosjo.net
> <mailto:jom at grosjo.net>> wrote:
> 
> The only point here of this fts-xapian is to get rid
> of solr (because it
> is just a nightmare to setup) and squat (because it is considere
> obsolete).
> 
> I already sent the changed in configure.ac, makefile.am, etc.. in order
> to include it in the dovecot, and it compiles properly
> 
> The only remaining point is to push it in hte git (yes, everything is
> already done)
> 
> On 2019-01-13 18:45, Aki Tuomi wrote:
> 
> On 13 January 2019 at 17:05 Joan Moreau via
> dovecot <dovecot at dovecot.org
> <mailto:dovecot at dovecot.org>> wrote:
> 
> Hi
> 
> Please find attached the beta release of FTS
> Xapian, with the objective
> to replace fts_squat that is being deprecated.
> 
> Configuration is exactly the same as for fts_squat:
> 
> plugin {
> 
> plugin = fts fts_xapian (...)
> fts = xapian
> fts_autoindex = yes
> fts_enforced = yes
> fts_xapian = partial=2 full=20
> 
> This is installed on my production server
> (>120Gb of mailboxes), and I
> will observe it during the coming days.
> 
> I will definitely appreciate that this is added in the core git of
> docevot, in order to have a versionning of
> it, to remove squat and let
> basic users able to avoid Solr alternative as much as possible.
> 
> Thanks
> 
> JM
> 
> Hi!
> 
> I still recommend you setup a, say, github
> repository for your plugin. We are not able to
> currently include your work in dovecot core as it
> is more work than just pushing the code into the
> repo. Maybe it can be included in the future.
> 
> If you want, I can help you in setting up the
> required configuration scripts and such to make
> it possible to compile it as plugin.
> 
> Then anyone can download it and install it for
> their dovecot, even if dovecot itself has been
> installed from packages, and also makes it
> possible for package maintainers to consider
> including it in distributions.
> 
> Aki

Paul Hecker
paul at iwascoding.com
http://www.iwascoding.com

iwascoding GmbH
Weserstr 18, 10247 Berlin
HRB 97269 B Amtsgericht Berlin-Charlottenburg
Geschäftsführer: Paul Hecker, Ilja Iwas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://dovecot.org/pipermail/dovecot/attachments/20190114/8c8ed457/attachment-0001.html>