[FTS Xapian] Beta release

Mon Jan 14 11:33:56 EET 2019

Difficult to figure out without a coredump + gdb 

I have also battled quite a lot to make sure dovecot can core dump on my
Archlinux servers. 

I remember that the key point was putting FS.SUID_DUMPABLE=2 in
/etc/sysctl.d/ conf files, LIMITCORE=INFINITY in
/etc/systemd/system/multi-user.target.wants/dovecot.service,  and
rebooting the server. 

My own coredumps are on /var/lib/systemd/coredump/ 

On 2019-01-14 10:20, Paul Hecker via dovecot wrote:

> Here it is: 
> 
> Jan 14 09:26:08 mail dovecot: indexer-worker(paul at iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Effective uid=8, gid=8, home=/var/spool/mail/iwascoding/paul
> Jan 14 09:26:08 mail dovecot: indexer-worker(paul at iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Quota root: name=User quota backend=dict args=:file:/var/spool/mail/iwascoding/paul/dovecot-quota
> Jan 14 09:26:08 mail dovecot: indexer-worker(paul at iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Quota rule: root=User quota mailbox=* bytes=2147483648 messages=0
> Jan 14 09:26:08 mail dovecot: indexer-worker(paul at iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Quota rule: root=User quota mailbox=* bytes=2147483648 messages=60000
> Jan 14 09:26:08 mail dovecot: indexer-worker(paul at iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Quota grace: root=User quota bytes=214748364 (10%)
> Jan 14 09:26:08 mail dovecot: indexer-worker(paul at iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: dict quota: user=paul at iwascoding.com, uri=file:/var/spool/mail/iwascoding/paul/dovecot-quota, noenforcing=0
> Jan 14 09:26:08 mail dovecot: indexer-worker(paul at iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Namespace inbox: type=private, prefix=, sep=/, inbox=yes, hidden=no, list=yes, subscriptions=yes location=mdbox:~/mdbox
> Jan 14 09:26:08 mail dovecot: indexer-worker(paul at iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: fs: root=/var/spool/mail/iwascoding/paul/mdbox, index=, indexpvt=, control=, inbox=, alt=
> Jan 14 09:26:08 mail dovecot: indexer-worker(paul at iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: FTS Xapian: Partial=2, Full=20 DB_PATH=/var/spool/mail/iwascoding/paul/mdbox/xapian-indexes
> Jan 14 09:26:08 mail dovecot: indexer-worker(paul at iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: quota: quota_over_flag check: quota_over_script unset - skipping
> Jan 14 09:26:08 mail dovecot: indexer-worker(paul at iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Mailbox sent: Mailbox opened because: indexing
> Jan 14 09:26:08 mail dovecot: indexer-worker(paul at iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: FTS Xapian : Mailbox sent : Last UID=0
> Jan 14 09:26:08 mail dovecot: indexer-worker(paul at iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: FTS Xapian : Mailbox sent : Last UID=0
> Jan 14 09:26:08 mail dovecot: indexer-worker(paul at iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Namespace : Using permissions from /var/spool/mail/iwascoding/paul/mdbox: mode=0700 gid=default
> Jan 14 09:26:08 mail dovecot: indexer-worker(paul at iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Mailbox sent: UID 1: Opened mail because: fts indexing
> Jan 14 09:26:08 mail dovecot: indexer-worker(paul at iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Opening RW /var/spool/mail/iwascoding/paul/mdbox/xapian-indexes/db_9ddfe10d8a8a8a568c120000654d370e
> Jan 14 09:26:08 mail dovecot: indexer-worker(paul at iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Mailbox sent: UID 2: Opened mail because: fts indexing
> Jan 14 09:26:08 mail dovecot: indexer-worker(paul at iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Mailbox sent: UID 3: Opened mail because: fts indexing
> 
> Thank you! 
> 
> On 14. Jan 2019, at 10:11, Joan Moreau via dovecot <dovecot at dovecot.org> wrote: 
> 
> Can you send the log part that includes the "init" of the plugins (something similar as below) ? 
> 
> WHich version of Xapian are you on ? 
> 
> Jan 14 09:10:04 gjserver dovecot[31082]: indexer-worker(admin at grosjo.net)<14725><apZfHexRPFy9QAAA0thIag:UL+TNOxRPFyFOQAA0thIag>: FTS Xapian: Partial=2, Full=20 DB_PATH=/data/mail/grosjo.net/admin/xapian-indexes [1]
> Jan 14 09:10:04 gjserver dovecot[31082]: indexer-worker(admin at grosjo.net)<14725><apZfHexRPFy9QAAA0thIag:UL+TNOxRPFyFOQAA0thIag>: FTS Xapian : Mailbox Mail : Last UID=815055
> Jan 14 09:10:04 gjserver dovecot[31082]: indexer-worker(admin at grosjo.net)<14725><apZfHexRPFy9QAAA0thIag:UL+TNOxRPFyFOQAA0thIag>: FTS Xapian : Mailbox Mail : Last UID=815055
> Jan 14 09:10:04 gjserver dovecot[31082]: indexer-worker(admin at grosjo.net)<14725><apZfHexRPFy9QAAA0thIag:UL+TNOxRPFyFOQAA0thIag>: Opening RW /data/mail/grosjo.net/admin/xapian-indexes/db_5c935034609bc14c0e550000d6a3092d [2] 
> 
> On 2019-01-14 10:08, Paul Hecker via dovecot wrote: 
> Hi,
> 
> I installed and tested your version, but the indexer process crashes reproducible with the following command after about 2000 messages were indexed:
> 
> doveadm index -u paul at iwascoding.com -q \*
> 
> Jan 14 09:26:15 mail dovecot: indexer-worker(paul at iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Debug: Mailbox sent: UID 2038: Opened mail because: fts indexing
> Jan 14 09:26:15 mail dovecot: indexer-worker: Error: terminate called after throwing an instance of 'std::bad_alloc'
> Jan 14 09:26:15 mail dovecot: indexer-worker: Error:   what():  std::bad_alloc
> Jan 14 09:26:15 mail dovecot: indexer: Error: Indexer worker disconnected, discarding 48 requests for paul at iwascoding.com
> Jan 14 09:26:15 mail dovecot: indexer-worker(paul at iwascoding.com)<16777><IKpfOqBHPFyJQQAADYqDFA>: Fatal: master: service(indexer-worker): child 16777 killed with signal 6 (core dumps disabled - https://dovecot.org/bugreport.html#coredumps)
> 
> I tried to delete the message, but this does not help (crashes e.g. after message 2029 or 2044). Other folders with fewer messages were successfully indexed before.
> 
> Sorry, could not convince dovecot to create core dumps (read the docs, changed /proc/sys/kernel/core_pattern, added LimitCORE=unlimited/infinity, even created /etc/systemd/system/dovecot.service.d/coredump.conf to no avail). Custom Dovecot 2.3.4 on Debian Stretch.
> 
> Thanks,
> Paul

Links:
------
[1] http://grosjo.net/admin/xapian-indexes
[2]
http://grosjo.net/admin/xapian-indexes/db_5c935034609bc14c0e550000d6a3092d
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://dovecot.org/pipermail/dovecot/attachments/20190114/f014cc73/attachment.html>