Hello Teemu,
I am working on this today.
The errors I have might be an artefact of the virtual machine I am using (KVM), especially the multiple snapshots. I will send a confirmation when I am sure.
However, I had some difficulties understanding the principle of the original script, and how it runs. Perhaps the documentation can be improved a little bit.
Kind regards, André
On 04/05/18 07:11, Teemu Huovila wrote:
Hello
Could you plese
- send the full output of doveconf -n
- Check the SOLR logs for any errors
- Describe your dovecot architecture, ie. if you are running a single backend or a more complex configuration.
- Provide a backtrace of the core dump using the instructions on https://dovecot.org/bugreport.html
br, Teemu
On 04.05.2018 01:13, André Rodier wrote:
On 02/05/18 22:17, André Rodier wrote:
On 02/05/18 11:45, André Rodier wrote:
On 2018-05-01 21:29, André Rodier wrote:
On 2018-05-01 07:22, André Rodier wrote:
Hello,
I am trying to use Doevecot fts, with solr the script provided.
To rebuild the index, I use the command: doveadm -D index -u mirina 'inbox'
To rescan, I use: doveadm -D fts rescan -u mirina
But when I do a search, with doveadm, the program hangs:
doveadm -D search -u mirina text Madagascar
> Debug: Loading modules from directory: /usr/lib/dovecot/modules > Debug: Module loaded: /usr/lib/dovecot/modules/lib10_quota_plugin.so > Debug: Module loaded: /usr/lib/dovecot/modules/lib20_fts_plugin.so > Debug: Module loaded: /usr/lib/dovecot/modules/lib21_fts_solr_plugin.so > Debug: Loading modules from directory: /usr/lib/dovecot/modules/doveadm > Debug: Skipping module doveadm_acl_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol: acl_lookup_dict_iterate_visible_next (this is usually intentional, so just ignore this message) > Debug: Skipping module doveadm_expire_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_expire_plugin.so: undefined symbol: expire_set_deinit (this is usually intentional, so just ignore this message) > Debug: Module loaded: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_quota_plugin.so > Debug: Module loaded: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_sieve_plugin.so > Debug: Skipping module doveadm_fts_lucene_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_lucene_plugin.so: undefined symbol: lucene_index_iter_deinit (this is usually intentional, so just ignore this m$ > ssage) > Debug: Module loaded: /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_plugin.so > Debug: Skipping module doveadm_mail_crypt_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/libdoveadm_mail_crypt_plugin.so: undefined symbol: mail_crypt_box_get_pvt_digests (this is usually intentional, so just ignore thi$ message) > doveadm(mirina): Debug: auth USER input: mirina home=/home/users/mirina uid=1002 gid=1001 mail=mirina@homebox.space > doveadm(mirina): Debug: Added userdb setting: mail=mirina@homebox.space doveadm(mirina): Debug: Effective uid=1002, gid=1001, home=/home/users/mirina > doveadm(mirina): Debug: Quota root: name=User quota backend=maildir args= > doveadm(mirina): Debug: Quota rule: root=User quota mailbox=* bytes=2147483648 messages=0 > doveadm(mirina): Debug: Quota grace: root=User quota bytes=214748364 (10%) > doveadm(mirina): Debug: Namespace inbox: type=private, prefix=, sep=/, inbox=yes, hidden=no, list=yes, subscriptions=yes location=maildir:~/mails/maildir:INDEX=~/mails/indexes/ > doveadm(mirina): Debug: maildir++: root=/home/users/mirina/mails/maildir, index=/home/users/mirina/mails/indexes, indexpvt=, control=, inbox=/home/users/mirina/mails/maildir, alt= > doveadm(mirina): Debug: quota: quota_over_flag check: STORAGE ret=1 value=134 limit=2097152 > doveadm(mirina): Debug: quota: quota_over_flag check: MESSAGE ret=0 value=3 limit=0 > doveadm(mirina): Debug: quota: quota_over_flag=0((null)) vs currently overquota=0 > doveadm(mirina): Debug: Namespace : Using permissions from /home/users/mirina/mails/maildir: mode=0700 gid=default > doveadm(mirina): Debug: http-client: host localhost: Host created > doveadm(mirina): Debug: http-client: host localhost: DNS lookup successful; got 2 IPs > doveadm(mirina): Debug: http-client: peer [::1]:8080: Peer created > doveadm(mirina): Debug: http-client: queue http://localhost:8080: Setting up connection to [::1]:8080 (1 requests pending) > doveadm(mirina): Debug: http-client: peer [::1]:8080: Linked queue http://localhost:8080 (1 queues linked) > doveadm(mirina): Debug: http-client: queue http://localhost:8080: Started new connection to [::1]:8080 > doveadm(mirina): Debug: http-client: request [Req1: GET http://localhost:8080/solr/select?fl=uid,score&rows=1&sort=uid+asc&q=%7b!lucene+q.op%3dAND%7d(hdr:Madagascar+OR+body:Madagascar)&fq=%2Bbox:838a1701af1ae65a3e12000032c62018+%2Buser:mi$ > ina]: Submitted > doveadm(mirina): Debug: http-client: Waiting for 1 requests to finish > doveadm(mirina): Debug: http-client: peer [::1]:8080: Creating 1 new connections to handle requests (already 0 usable, connecting to 0, closing 0) > doveadm(mirina): Debug: http-client: peer [::1]:8080: Making new connection 1 of 1 > doveadm(mirina): Debug: http-client: conn [::1]:8080 [0]: HTTP connection created (1 parallel connections exist) > doveadm(mirina): Debug: http-client: conn [::1]:8080 [0]: Connected > doveadm(mirina): Debug: http-client: conn [::1]:8080 [0]: Ready for requests > doveadm(mirina): Debug: http-client: peer [::1]:8080: Successfully connected (connections=1) > doveadm(mirina): Debug: http-client: peer [::1]:8080: Using 1 idle connections to handle 1 requests (1 total connections ready) > doveadm(mirina): Debug: http-client: queue http://localhost:8080: Connection to peer [::1]:8080 claimed request [Req1: GET http://localhost:8080/solr/select?fl=uid,score&rows=1&sort=uid+asc&q=%7b!lucene+q.op%3dAND%7d(hdr:Madagascar+OR+bod$ > :Madagascar)&fq=%2Bbox:838a1701af1ae65a3e12000032c62018+%2Buser:mirina] > doveadm(mirina): Debug: http-client: conn [::1]:8080 [0]: Claimed request [Req1: GET http://localhost:8080/solr/select?fl=uid,score&rows=1&sort=uid+asc&q=%7b!lucene+q.op%3dAND%7d(hdr:Madagascar+OR+body:Madagascar)&fq=%2Bbox:838a1701af1ae6$ > a3e12000032c62018+%2Buser:mirina] > [...] > doveadm(mirina): Debug: http-client: request [Req5: GET http://localhost:8080/solr/select?fl=uid,score&rows=1&sort=uid+asc&q=%7b!lucene+q.op%3dAND%7d(hdr:Madagascar+OR+body:Madagascar)&fq=%2Bbox:6dd14d3aae1ae65a3e12000032c62018+%2Buser:mirina]: Sent header > doveadm(mirina): Debug: http-client: peer [::1]:8080: No more requests to service for this peer (1 connections exist) > doveadm(mirina): Debug: http-client: conn [::1]:8080 [0]: Got 200 response for request [Req5: GET http://localhost:8080/solr/select?fl=uid,score&rows=1&sort=uid+asc&q=%7b!lucene+q.op%3dAND%7d(hdr:Madagascar+OR+body:Madagascar)&fq=%2Bbox:6dd14d3aae1ae65a3e12000032c62018+%2Buser:mirina] (took 2 ms + 0 ms in queue) > doveadm(mirina): Debug: http-client: conn [::1]:8080 [0]: Response payload stream destroyed (0 ms after initial response) > doveadm(mirina): Debug: http-client: request [Req5: GET http://localhost:8080/solr/select?fl=uid,score&rows=1&sort=uid+asc&q=%7b!lucene+q.op%3dAND%7d(hdr:Madagascar+OR+body:Madagascar)&fq=%2Bbox:6dd14d3aae1ae65a3e12000032c62018+%2Buser:mirina]: Finished > doveadm(mirina): Debug: http-client: queue http://localhost:8080: Dropping request [Req5: GET http://localhost:8080/solr/select?fl=uid,score&rows=1&sort=uid+asc&q=%7b!lucene+q.op%3dAND%7d(hdr:Madagascar+OR+body:Madagascar)&fq=%2Bbox:6dd14d3aae1ae65a3e12000032c62018+%2Buser:mirina] > doveadm(mirina): Debug: http-client: host localhost: Host is idle (timeout = 1799974 msecs) > doveadm(mirina): Debug: http-client: request [Req5: GET http://localhost:8080/solr/select?fl=uid,score&rows=1&sort=uid+asc&q=%7b!lucene+q.op%3dAND%7d(hdr:Madagascar+OR+body:Madagascar)&fq=%2Bbox:6dd14d3aae1ae65a3e12000032c62018+%2Buser:mirina]: Free (requests left=1) > doveadm(mirina): Debug: http-client: All requests finished
When I do the search with an imap client, I have this error after a long timeout:
> May 01 07:08:58 homebox dovecot[5683]: imap(mirina): Panic: file imap-client.c: line 854 (client_check_command_hangs): assertion failed: (client->io != NULL || (client->output_cmd_lock != NULL && client->output_cmd_lock != client->input_lock)) > May 01 07:08:58 homebox dovecot[5683]: imap(mirina): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x95e92) [0x7f216b6f6e92] -> /usr/lib/dovecot/libdovecot.so.0(+0x95f8d) [0x7f216b6f6f8d] -> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7f216b68ca91] -> dovecot/imap(client_continue_pending_input+0x1fc) [0x55f44a7eb15c] -> dovecot/imap(client_input+0x120) [0x55f44a7eb460] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x52) [0x7f216b70b9f2] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x109) [0x7f216b70d029] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x3c) [0x7f216b70ba8c] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7f216b70bc38] -> /usr/lib/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7f216b692fd3] -> dovecot/imap(main+0x328) [0x55f44a7dde68] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1) [0x7f216b2e22e1] -> dovecot/imap(_start+0x2a) [0x55f44a7ddfea] > May 01 07:08:58 homebox dovecot[5683]: imap(mirina): Fatal: master: service(imap): child 6149 killed with signal 6 (core dumps disabled) > May 01 07:08:58 homebox in.imapproxyd[20305]: Raw_Proxy(): IMAP server unexpectedly closed the connection on sd 11
My 10-master.conf:
service decode2text { executable = script /usr/local/bin/decode2text.sh user = dovecot unix_listener decode2text { mode = 0666 } }
And my 90-plugins.cof: # Add dovecot fts config fts = solr fts_autoindex = no fts_solr = url=http://localhost:8080/solr/ debug fts_decoder = decode2text
Am I doing anything wrong?
Thanks for your help.
André
hello again,
The issue is not solved, but I have been able to see an error message: indexer-worker(andre): Error: parser script didn't send any data
This is weird, because 1) this error is intermittent, 2) I explicitly modified the script to echo a specific word in al the cases.
Any idea, someone?
Thanks
Hello again,
I am still having this issue.
The inbox I am testing has only three emails, with three attachments. The same document in doc, pdf and odt...
Thanks for your help and advice, if you have any.
Kind regards, André
Hello again, I really hope to find some Dovecot support on this list.
On Debian stretch, I have tried versions from stable (2.2.27-3+deb9u2) and from the backports (2.2.34-2~bpo9+1)
None of the version works. The operation doveadm search, or by imap is frozen...Even if I use the exact script provided by Dovecot decode2txt.sh.
Any idea from the Dovecot core team would be nice.
Thanks. André Rodier.
Hello - again,
I can only find a few explanations for the lack of answers to my questions..
If nobody knows how the dovecot fts code works anymore, it's OK, just just tell, so I stop searching what I am doing wrong...
If Dovecot team is planing to push another fts engine, it's OK as well, but I need to know...
I am loosing a lot of time on this, and I start to be upset.
To summarise:
- I have a custom script that extract text from attachments, for instance doc or pdf files.
- I successfully tested my script on the command line, and it returns the right text every time, in a few ms. I use a one page doc or pdf, with just a paragraph.
- When using the dovecot search function with the same documents attached to an email, I have a timeout, and sometimes an error message.
My dovecot plugin configuration:
fts = solr fts_solr = url=http://localhost:8080/solr/ fts_decoder = decode2text fts_index_timeout = 10
The script is a non modified version of decode2text provided by the Dovecot installation.
Please, just point me in some direction.
André Rodier.