[FTS Xapian] Beta release
Paul Hecker
paul at iwascoding.com
Mon Jan 14 11:19:57 EET 2019
Here it is:
Jan 14 09:26:08 mail dovecot: indexer-worker(paul at iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Effective uid=8, gid=8, home=/var/spool/mail/iwascoding/paul
Jan 14 09:26:08 mail dovecot: indexer-worker(paul at iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Quota root: name=User quota backend=dict args=:file:/var/spool/mail/iwascoding/paul/dovecot-quota
Jan 14 09:26:08 mail dovecot: indexer-worker(paul at iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Quota rule: root=User quota mailbox=* bytes=2147483648 messages=0
Jan 14 09:26:08 mail dovecot: indexer-worker(paul at iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Quota rule: root=User quota mailbox=* bytes=2147483648 messages=60000
Jan 14 09:26:08 mail dovecot: indexer-worker(paul at iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Quota grace: root=User quota bytes=214748364 (10%)
Jan 14 09:26:08 mail dovecot: indexer-worker(paul at iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: dict quota: user=paul at iwascoding.com, uri=file:/var/spool/mail/iwascoding/paul/dovecot-quota, noenforcing=0
Jan 14 09:26:08 mail dovecot: indexer-worker(paul at iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Namespace inbox: type=private, prefix=, sep=/, inbox=yes, hidden=no, list=yes, subscriptions=yes location=mdbox:~/mdbox
Jan 14 09:26:08 mail dovecot: indexer-worker(paul at iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: fs: root=/var/spool/mail/iwascoding/paul/mdbox, index=, indexpvt=, control=, inbox=, alt=
Jan 14 09:26:08 mail dovecot: indexer-worker(paul at iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: FTS Xapian: Partial=2, Full=20 DB_PATH=/var/spool/mail/iwascoding/paul/mdbox/xapian-indexes
Jan 14 09:26:08 mail dovecot: indexer-worker(paul at iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: quota: quota_over_flag check: quota_over_script unset - skipping
Jan 14 09:26:08 mail dovecot: indexer-worker(paul at iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Mailbox sent: Mailbox opened because: indexing
Jan 14 09:26:08 mail dovecot: indexer-worker(paul at iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: FTS Xapian : Mailbox sent : Last UID=0
Jan 14 09:26:08 mail dovecot: indexer-worker(paul at iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: FTS Xapian : Mailbox sent : Last UID=0
Jan 14 09:26:08 mail dovecot: indexer-worker(paul at iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Namespace : Using permissions from /var/spool/mail/iwascoding/paul/mdbox: mode=0700 gid=default
Jan 14 09:26:08 mail dovecot: indexer-worker(paul at iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Mailbox sent: UID 1: Opened mail because: fts indexing
Jan 14 09:26:08 mail dovecot: indexer-worker(paul at iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Opening RW /var/spool/mail/iwascoding/paul/mdbox/xapian-indexes/db_9ddfe10d8a8a8a568c120000654d370e
Jan 14 09:26:08 mail dovecot: indexer-worker(paul at iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Mailbox sent: UID 2: Opened mail because: fts indexing
Jan 14 09:26:08 mail dovecot: indexer-worker(paul at iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Mailbox sent: UID 3: Opened mail because: fts indexing
Thank you!
> On 14. Jan 2019, at 10:11, Joan Moreau via dovecot <dovecot at dovecot.org> wrote:
>
> Can you send the log part that includes the "init" of the plugins (something similar as below) ?
>
> WHich version of Xapian are you on ?
>
> Jan 14 09:10:04 gjserver dovecot[31082]: indexer-worker(admin at grosjo.net)<14725><apZfHexRPFy9QAAA0thIag:UL+TNOxRPFyFOQAA0thIag>: FTS Xapian: Partial=2, Full=20 DB_PATH=/data/mail/grosjo.net/admin/xapian-indexes
> Jan 14 09:10:04 gjserver dovecot[31082]: indexer-worker(admin at grosjo.net)<14725><apZfHexRPFy9QAAA0thIag:UL+TNOxRPFyFOQAA0thIag>: FTS Xapian : Mailbox Mail : Last UID=815055
> Jan 14 09:10:04 gjserver dovecot[31082]: indexer-worker(admin at grosjo.net)<14725><apZfHexRPFy9QAAA0thIag:UL+TNOxRPFyFOQAA0thIag>: FTS Xapian : Mailbox Mail : Last UID=815055
> Jan 14 09:10:04 gjserver dovecot[31082]: indexer-worker(admin at grosjo.net)<14725><apZfHexRPFy9QAAA0thIag:UL+TNOxRPFyFOQAA0thIag>: Opening RW /data/mail/grosjo.net/admin/xapian-indexes/db_5c935034609bc14c0e550000d6a3092d
>
>
>
>
>
>
> On 2019-01-14 10:08, Paul Hecker via dovecot wrote:
>
>> Hi,
>>
>> I installed and tested your version, but the indexer process crashes reproducible with the following command after about 2000 messages were indexed:
>>
>> doveadm index -u paul at iwascoding.com <mailto:paul at iwascoding.com> -q \*
>>
>> Jan 14 09:26:15 mail dovecot: indexer-worker(paul at iwascoding.com <mailto:paul at iwascoding.com>)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Mailbox sent: UID 2038: Opened mail because: fts indexing
>> Jan 14 09:26:15 mail dovecot: indexer-worker: Error: terminate called after throwing an instance of 'std::bad_alloc'
>> Jan 14 09:26:15 mail dovecot: indexer-worker: Error: what(): std::bad_alloc
>> Jan 14 09:26:15 mail dovecot: indexer: Error: Indexer worker disconnected, discarding 48 requests for paul at iwascoding.com <mailto:paul at iwascoding.com>
>> Jan 14 09:26:15 mail dovecot: indexer-worker(paul at iwascoding.com <mailto:paul at iwascoding.com>)<16777><IKpfOqBHPFyJQQAADYqDFA>: Fatal: master: service(indexer-worker): child 16777 killed with signal 6 (core dumps disabled - https://dovecot.org/bugreport.html#coredumps <https://dovecot.org/bugreport.html#coredumps>)
>>
>> I tried to delete the message, but this does not help (crashes e.g. after message 2029 or 2044). Other folders with fewer messages were successfully indexed before.
>>
>> Sorry, could not convince dovecot to create core dumps (read the docs, changed /proc/sys/kernel/core_pattern, added LimitCORE=unlimited/infinity, even created /etc/systemd/system/dovecot.service.d/coredump.conf to no avail). Custom Dovecot 2.3.4 on Debian Stretch.
>>
>> Thanks,
>> Paul
>>
>>
>>> On 14. Jan 2019, at 07:42, Joan Moreau via dovecot <dovecot at dovecot.org <mailto:dovecot at dovecot.org>> wrote:
>>>
>>> Thank you Stephan.
>>>
>>> The version here shall be up and running : https://github.com/grosjo/fts-xapian <https://github.com/grosjo/fts-xapian>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On 2019-01-14 00:07, Stephan Bosch wrote:
>>>
>>>>
>>>>
>>>> Op 13/01/2019 om 21:25 schreef Joan Moreau via dovecot:
>>>>>
>>>>>
>>>>> I tried to combined it, the "autoreconf" errors are solved
>>>>>
>>>>> Now, when I type "make install", the lib is not pushed into dovecot folder, but somewhere in /usr/local/...
>>>>>
>>>>> How to adjust this to have it arriving in the proper folder ?
>>>>>
>>>>
>>>> Depends on your system. It mostly a matter of setting a proper --prefix directory for configure, but other paths are configurable as well. I usually check what the official distribution package for Dovecot is doing and use that as a basis.
>>>>
>>>> For Debian I use the following configure command:
>>>>
>>>> ./configure --with-ldap=plugin --with-ssl=openssl --with-sql=plugin --with-lua=plugin --with-pgsql --with-mysql --with-sqlite \
>>>> --with-gssapi=plugin --with-solr --with-ioloop=best --enable-maintainer-mode \
>>>> --prefix=/usr --sysconfdir=/etc --libexecdir=/usr/lib --localstatedir=/var --mandir=/usr/share/man \
>>>> --infodir=/usr/share/info --with-moduledir=/usr/lib/dovecot/modules --disable-rpath --disable-static
>>>>
>>>> Regards,
>>>>
>>>> Stephan
>>>>
>>>>>
>>>>> On 2019-01-13 21:01, Tuomi, Aki wrote:
>>>>>
>>>>>> You copied your Makefile.am there. Stephan made you a working version, can you try that?
>>>>>> (sorry for dup)
>>>>>> Aki
>>>>>> -------- Original message --------
>>>>>> From: Joan Moreau <jom at grosjo.net <mailto:jom at grosjo.net>>
>>>>>> Date: 13/01/2019 21:39 (GMT+02:00)
>>>>>> To: Stephan Bosch <stephan at rename-it.nl <mailto:stephan at rename-it.nl>>
>>>>>> Cc: Aki Tuomi <aki.tuomi at open-xchange.com <mailto:aki.tuomi at open-xchange.com>>
>>>>>> Subject: Re: [FTS Xapian] Beta release
>>>>>>
>>>>>> I used the skeleton from Aki : https://github.com/grosjo/fts-xapian <https://github.com/grosjo/fts-xapian>
>>>>>>
>>>>>> However, when I try to act as a visitor, I reach teh follwoing error:
>>>>>>
>>>>>> # autoreconf -vi
>>>>>> autoreconf: Entering directory `.'
>>>>>> autoreconf: configure.ac: not using Gettext
>>>>>> autoreconf: running: aclocal -I m4
>>>>>> autoreconf: configure.ac: tracing
>>>>>> autoreconf: running: libtoolize --copy
>>>>>> libtoolize: putting auxiliary files in AC_CONFIG_AUX_DIR, '.'.
>>>>>> libtoolize: copying file './ltmain.sh'
>>>>>> libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'm4'.
>>>>>> libtoolize: copying file 'm4/libtool.m4'
>>>>>> libtoolize: copying file 'm4/ltoptions.m4'
>>>>>> libtoolize: copying file 'm4/ltsugar.m4'
>>>>>> libtoolize: copying file 'm4/ltversion.m4'
>>>>>> libtoolize: copying file 'm4/lt~obsolete.m4'
>>>>>> autoreconf: running: /usr/bin/autoconf
>>>>>> autoreconf: running: /usr/bin/autoheader
>>>>>> autoreconf: running: automake --add-missing --copy --no-force
>>>>>> configure.ac:9: installing './compile'
>>>>>> configure.ac:11: installing './config.guess'
>>>>>> configure.ac:11: installing './config.sub'
>>>>>> configure.ac:7: installing './install-sh'
>>>>>> configure.ac:7: installing './missing'
>>>>>> src/Makefile.am: installing './depcomp'
>>>>>> /usr/share/automake-1.16/am/depend2.am: error: am__fastdepCXX does not appear in AM_CONDITIONAL
>>>>>> /usr/share/automake-1.16/am/depend2.am: The usual way to define 'am__fastdepCXX' is to add 'AC_PROG_CXX'
>>>>>> /usr/share/automake-1.16/am/depend2.am: to 'configure.ac' and run 'aclocal' and 'autoconf' again
>>>>>> src/Makefile.am: error: C++ source seen but 'CXX' is undefined
>>>>>> src/Makefile.am: The usual way to define 'CXX' is to add 'AC_PROG_CXX'
>>>>>> src/Makefile.am: to 'configure.ac' and run 'autoconf' again.
>>>>>> src/Makefile.am:11: warning: variable 'NOPLUGIN_LDFLAGS' is defined but no program or
>>>>>> src/Makefile.am:11: library has 'NOPLUGIN' as canonical name (possible typo)
>>>>>> autoreconf: automake failed with exit status: 1
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 2019-01-13 20:24, Stephan Bosch wrote:
>>>>>>
>>>>>> Oh, right, a distribution tarball doesn't include some of the
>>>>>> necessary files for your repository like autogen.sh and
>>>>>> .gitignore. The attached tarball includes all those and is ready
>>>>>> for `git init`. The previous tarball was made with `make
>>>>>> distcheck` from this one.
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Stephan.
>>>>>>
>>>>>> Op 13/01/2019 om 20:14 schreef Stephan Bosch:
>>>>>>
>>>>>> Hi Joan,
>>>>>>
>>>>>> Op 13/01/2019 om 19:03 schreef Aki Tuomi:
>>>>>>
>>>>>> Yes, from compiling point of view it is done.
>>>>>>
>>>>>> Unfortunately what is not done is all the other work
>>>>>> involved, such as fixing all the inevitable bugs it has
>>>>>> and maintaining it. We do not want, at this moment, take
>>>>>> up maintaining and developing yet another FTS plugin as
>>>>>> we have plenty of things to do already.
>>>>>>
>>>>>> I invite you to setup your own repository and provide
>>>>>> this plugin from there, being the maintainer of this
>>>>>> plugin. We can add a link to your plugin on our FTS page
>>>>>> so people can also find it.
>>>>>>
>>>>>> There are other plugins like this, e.g.
>>>>>> https://github.com/st3fan/dovecot-xaps-plugin <https://github.com/st3fan/dovecot-xaps-plugin>
>>>>>>
>>>>>>
>>>>>> I turned the code you provided into a separate plugin
>>>>>> package. The distribution tarball is attached.
>>>>>>
>>>>>> Notable changes:
>>>>>>
>>>>>> - Added example copyright headers and COPYING and AUTHORS
>>>>>> files. You should modify those to your preference.
>>>>>> - Added README and INSTALL files (in markdown using Pandoc).
>>>>>> Those need to be amended with details.
>>>>>> - Amended the plugin code to display a debug message with the
>>>>>> plugin name and version upon plugin load.
>>>>>>
>>>>>> I advise you to turn this into a git repository and continue from there.
>>>>>>
>>>>>> I do not recommend releasing this plugin with the
>>>>>> -fpermissive flag and the resulting warning as it is now. But
>>>>>> I'm assuming this is still a work in progress, so that is OK.
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Stephan.
>>>>>>
>>>>>>
>>>>>> On 13 January 2019 at 19:52 Joan Moreau <jom at grosjo.net <mailto:jom at grosjo.net>
>>>>>> <mailto:jom at grosjo.net <mailto:jom at grosjo.net>>> wrote:
>>>>>>
>>>>>>
>>>>>> The only point here of this fts-xapian is to get rid
>>>>>> of solr (because it
>>>>>> is just a nightmare to setup) and squat (because it is considere
>>>>>> obsolete).
>>>>>>
>>>>>> I already sent the changed in configure.ac, makefile.am, etc.. in order
>>>>>> to include it in the dovecot, and it compiles properly
>>>>>>
>>>>>> The only remaining point is to push it in hte git (yes, everything is
>>>>>> already done)
>>>>>>
>>>>>> On 2019-01-13 18:45, Aki Tuomi wrote:
>>>>>>
>>>>>> On 13 January 2019 at 17:05 Joan Moreau via
>>>>>> dovecot <dovecot at dovecot.org <mailto:dovecot at dovecot.org>
>>>>>> <mailto:dovecot at dovecot.org <mailto:dovecot at dovecot.org>>> wrote:
>>>>>>
>>>>>> Hi
>>>>>>
>>>>>> Please find attached the beta release of FTS
>>>>>> Xapian, with the objective
>>>>>> to replace fts_squat that is being deprecated.
>>>>>>
>>>>>> Configuration is exactly the same as for fts_squat:
>>>>>>
>>>>>> plugin {
>>>>>>
>>>>>> plugin = fts fts_xapian (...)
>>>>>> fts = xapian
>>>>>> fts_autoindex = yes
>>>>>> fts_enforced = yes
>>>>>> fts_xapian = partial=2 full=20
>>>>>>
>>>>>> This is installed on my production server
>>>>>> (>120Gb of mailboxes), and I
>>>>>> will observe it during the coming days.
>>>>>>
>>>>>> I will definitely appreciate that this is added in the core git of
>>>>>> docevot, in order to have a versionning of
>>>>>> it, to remove squat and let
>>>>>> basic users able to avoid Solr alternative as much as possible.
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> JM
>>>>>>
>>>>>> Hi!
>>>>>>
>>>>>> I still recommend you setup a, say, github
>>>>>> repository for your plugin. We are not able to
>>>>>> currently include your work in dovecot core as it
>>>>>> is more work than just pushing the code into the
>>>>>> repo. Maybe it can be included in the future.
>>>>>>
>>>>>> If you want, I can help you in setting up the
>>>>>> required configuration scripts and such to make
>>>>>> it possible to compile it as plugin.
>>>>>>
>>>>>> Then anyone can download it and install it for
>>>>>> their dovecot, even if dovecot itself has been
>>>>>> installed from packages, and also makes it
>>>>>> possible for package maintainers to consider
>>>>>> including it in distributions.
>>>>>>
>>>>>> Aki
>>>>>>
>>>>
>>>>
>>
>>
>> Paul Hecker
>> paul at iwascoding.com <mailto:paul at iwascoding.com>
>> http://www.iwascoding.com <http://www.iwascoding.com/>
>>
>> iwascoding GmbH
>> Weserstr 18, 10247 Berlin
>> HRB 97269 B Amtsgericht Berlin-Charlottenburg
>> Geschäftsführer: Paul Hecker, Ilja Iwas
>>
>>
>>
>>
>>
>>
>>
>>
Paul Hecker
paul at iwascoding.com <mailto:paul at iwascoding.com>
http://www.iwascoding.com <http://www.iwascoding.com/>
iwascoding GmbH
Weserstr 18, 10247 Berlin
HRB 97269 B Amtsgericht Berlin-Charlottenburg
Geschäftsführer: Paul Hecker, Ilja Iwas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://dovecot.org/pipermail/dovecot/attachments/20190114/70ee63d5/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4127 bytes
Desc: not available
URL: <https://dovecot.org/pipermail/dovecot/attachments/20190114/70ee63d5/attachment-0001.p7s>
More information about the dovecot
mailing list