My Solr FTS problem
I am testing the solr FTS following the guide here: http://things.m31.ch/?p=379
Now I am having problem when I try to test:
# doveadm -v index -u fail2ban@mydomain.com Inbox doveadm(fail2ban@mydomain.com): Error: fts_solr: Invalid XML input at 1:0: not well-formed (invalid token) (near: { "responseHeader":{ "status":0, "QTime":0, "params":{ "q":"box:8864fa1d51ea1d5a7b12000096a1aaf8 AND user:fa) doveadm(fail2ban@mydomain.com): Error: Mailbox INBOX: Status lookup failed: Internal error occurred. Refer to server log for more information. [2017-11-30 13:05:24]
My system is CentOS 7, Dovecot 3.2.4 (installed from source with solr support), Java OpenJDK-1.8.0, Solr v7.1.0.
Could some one help me on this please?
Here is detailed debug out put:
[root@mail ~]# doveadm -vD index -u fail2ban@mydomain.com Inbox Debug: Loading modules from directory: /usr/lib64/dovecot Debug: Module loaded: /usr/lib64/dovecot/lib10_quota_plugin.so Debug: Module loaded: /usr/lib64/dovecot/lib20_fts_plugin.so Debug: Module loaded: /usr/lib64/dovecot/lib21_fts_solr_plugin.so Debug: Loading modules from directory: /usr/lib64/dovecot/doveadm Debug: Skipping module doveadm_acl_plugin, because dlopen() failed: /usr/lib64/dovecot/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol: acl_user_module (this is usually intentional, so just ignore this message) Debug: Skipping module doveadm_expire_plugin, because dlopen() failed: /usr/lib64/dovecot/doveadm/lib10_doveadm_expire_plugin.so: undefined symbol: expire_set_deinit (this is usually intentional, so just ignore this message) Debug: Module loaded: /usr/lib64/dovecot/doveadm/lib10_doveadm_quota_plugin.so Debug: Module loaded: /usr/lib64/dovecot/doveadm/lib10_doveadm_sieve_plugin.so Debug: Module loaded: /usr/lib64/dovecot/doveadm/lib20_doveadm_fts_plugin.so Debug: Skipping module doveadm_mail_crypt_plugin, because dlopen() failed: /usr/lib64/dovecot/doveadm/libdoveadm_mail_crypt_plugin.so: undefined symbol: mail_crypt_box_get_pvt_digests (this is usually intentional, so just ignore this message) doveadm(fail2ban@mydomain.com): Debug: Added userdb setting: mail=maildir:/home/vmail/mydomain.com/fail2ban/ doveadm(fail2ban@mydomain.com): Debug: Added userdb setting: plugin/quota_rule=*:bytes=0 doveadm(fail2ban@mydomain.com): Debug: Effective uid=5000, gid=5000, home=/home/vmail/mydomain.com/fail2ban/ doveadm(fail2ban@mydomain.com): Debug: Quota root: name=User quota backend=maildir args= doveadm(fail2ban@mydomain.com): Debug: Quota rule: root=User quota mailbox=* bytes=0 messages=0 doveadm(fail2ban@mydomain.com): Debug: Quota rule: root=User quota mailbox=Trash bytes=+104857600 messages=0 doveadm(fail2ban@mydomain.com): Debug: Quota warning: bytes=0 (90%) messages=0 reverse=no command=quota-warning 90 fail2ban@mydomain.com mydomain.com doveadm(fail2ban@mydomain.com): Debug: Quota warning: bytes=0 (80%) messages=0 reverse=no command=quota-warning 80 fail2ban@mydomain.com mydomain.com doveadm(fail2ban@mydomain.com): Debug: Quota grace: root=User quota bytes=0 (10%) doveadm(fail2ban@mydomain.com): Debug: Namespace inbox: type=private, prefix=, sep=, inbox=yes, hidden=no, list=yes, subscriptions=yes location=maildir:/home/vmail/mydomain.com/fail2ban/ doveadm(fail2ban@mydomain.com): Debug: maildir++: root=/home/vmail/mydomain.com/fail2ban, index=, indexpvt=, control=, inbox=/home/vmail/mydomain.com/fail2ban, alt= doveadm(fail2ban@mydomain.com): Debug: quota: quota_over_flag check: quota_over_script unset - skipping doveadm(fail2ban@mydomain.com): Debug: INBOX: Mailbox opened because: index doveadm(fail2ban@mydomain.com): Debug: http-client: host 127.0.0.1: Host created doveadm(fail2ban@mydomain.com): Debug: http-client: peer 127.0.0.1:8983: Peer created doveadm(fail2ban@mydomain.com): Debug: http-client: queue http://127.0.0.1:8983: Setting up connection to 127.0.0.1:8983 (1 requests pending) doveadm(fail2ban@mydomain.com): Debug: http-client: peer 127.0.0.1:8983: Linked queue http://127.0.0.1:8983 (1 queues linked) doveadm(fail2ban@mydomain.com): Debug: http-client: queue http://127.0.0.1:8983: Started new connection to 127.0.0.1:8983 doveadm(fail2ban@mydomain.com): Debug: http-client: request [Req1: GET http://127.0.0.1:8983/solr/dovecot/select?fl=uid&rows=1&sort=uid+desc&q=box:8864fa1d51ea1d5a7b12000096a1aaf8+AND+user:fail2ban@mydomain.com]: Submitted doveadm(fail2ban@mydomain.com): Debug: http-client: Waiting for 1 requests to finish doveadm(fail2ban@mydomain.com): Debug: http-client: peer 127.0.0.1:8983: Creating 1 new connections to handle requests (already 0 usable, connecting to 0, closing 0) doveadm(fail2ban@mydomain.com): Debug: http-client: peer 127.0.0.1:8983: Making new connection 1 of 1 doveadm(fail2ban@mydomain.com): Debug: http-client: conn 127.0.0.1:8983 [0]: HTTP connection created (1 parallel connections exist) doveadm(fail2ban@mydomain.com): Debug: http-client: conn 127.0.0.1:8983 [0]: Connected doveadm(fail2ban@mydomain.com): Debug: http-client: conn 127.0.0.1:8983 [0]: Ready for requests doveadm(fail2ban@mydomain.com): Debug: http-client: peer 127.0.0.1:8983: Successfully connected (connections=1) doveadm(fail2ban@mydomain.com): Debug: http-client: peer 127.0.0.1:8983: Using 1 idle connections to handle 1 requests (1 total connections ready) doveadm(fail2ban@mydomain.com): Debug: http-client: queue http://127.0.0.1:8983: Connection to peer 127.0.0.1:8983 claimed request [Req1: GET http://127.0.0.1:8983/solr/dovecot/select?fl=uid&rows=1&sort=uid+desc&q=box:8864fa1d51ea1d5a7b12000096a1aaf8+AND+user:fail2ban@mydomain.com]
doveadm(fail2ban@mydomain.com): Debug: http-client: conn 127.0.0.1:8983 [0]: Claimed request [Req1: GET http://127.0.0.1:8983/solr/dovecot/select?fl=uid&rows=1&sort=uid+desc&q=box:8864fa1d51ea1d5a7b12000096a1aaf8+AND+user:fail2ban@mydomain.com] doveadm(fail2ban@mydomain.com): Debug: http-client: request [Req1: GET http://127.0.0.1:8983/solr/dovecot/select?fl=uid&rows=1&sort=uid+desc&q=box:8864fa1d51ea1d5a7b12000096a1aaf8+AND+user:fail2ban@mydomain.com]: Sent header doveadm(fail2ban@mydomain.com): Debug: http-client: peer 127.0.0.1:8983: No more requests to service for this peer (1 connections exist) doveadm(fail2ban@mydomain.com): Debug: http-client: conn 127.0.0.1:8983 [0]: Got 200 response for request [Req1: GET http://127.0.0.1:8983/solr/dovecot/select?fl=uid&rows=1&sort=uid+desc&q=box:8864fa1d51ea1d5a7b12000096a1aaf8+AND+user:fail2ban@mydomain.com] (took 3 ms + 2 ms in queue) doveadm(fail2ban@mydomain.com): Error: fts_solr: Invalid XML input at 1:0: not well-formed (invalid token) (near: { "responseHeader":{ "status":0, "QTime":1, "params":{ "q":"box:8864fa1d51ea1d5a7b12000096a1aaf8 AND user:fa) doveadm(fail2ban@mydomain.com): Debug: http-client: conn 127.0.0.1:8983 [0]: Response payload stream destroyed (0 ms after initial response) doveadm(fail2ban@mydomain.com): Debug: http-client: request [Req1: GET http://127.0.0.1:8983/solr/dovecot/select?fl=uid&rows=1&sort=uid+desc&q=box:8864fa1d51ea1d5a7b12000096a1aaf8+AND+user:fail2ban@mydomain.com]: Finished doveadm(fail2ban@mydomain.com): Debug: http-client: queue http://127.0.0.1:8983: Dropping request [Req1: GET http://127.0.0.1:8983/solr/dovecot/select?fl=uid&rows=1&sort=uid+desc&q=box:8864fa1d51ea1d5a7b12000096a1aaf8+AND+user:fail2ban@mydomain.com] doveadm(fail2ban@mydomain.com): Debug: http-client: host 127.0.0.1: Host is idle (timeout = 100 msecs) doveadm(fail2ban@mydomain.com): Debug: http-client: request [Req1: GET http://127.0.0.1:8983/solr/dovecot/select?fl=uid&rows=1&sort=uid+desc&q=box:8864fa1d51ea1d5a7b12000096a1aaf8+AND+user:fail2ban@mydomain.com]: Free (requests left=1) doveadm(fail2ban@mydomain.com): Debug: http-client: All requests finished doveadm(fail2ban@mydomain.com): Error: Mailbox INBOX: Status lookup failed: Internal error occurred. Refer to server log for more information. [2017-11-30 13:13:57] doveadm(fail2ban@mydomain.com): Debug: http-client: peer 127.0.0.1:8983: Peer close doveadm(fail2ban@mydomain.com): Debug: http-client: peer 127.0.0.1:8983: Peer disconnect doveadm(fail2ban@mydomain.com): Debug: http-client: conn 127.0.0.1:8983 [0]: Peer closed doveadm(fail2ban@mydomain.com): Debug: http-client: conn 127.0.0.1:8983 [0]: Connection disconnect doveadm(fail2ban@mydomain.com): Debug: http-client: conn 127.0.0.1:8983 [0]: Connection destroy doveadm(fail2ban@mydomain.com): Debug: http-client: peer 127.0.0.1:8983: Peer destroy doveadm(fail2ban@mydomain.com): Debug: http-client: host 127.0.0.1: Host destroy doveadm(fail2ban@mydomain.com): Debug: http-client: queue http://127.0.0.1:8983: Destroy
Am 30. November 2017 22:14:59 MEZ schrieb Gao gao@pztop.com:
I am testing the solr FTS following the guide here: http://things.m31.ch/?p=379
Now I am having problem when I try to test:
# doveadm -v index -u fail2ban@mydomain.com Inbox doveadm(fail2ban@mydomain.com): Error: fts_solr: Invalid XML input at 1:0: not well-formed (invalid token) (near: { "responseHeader":{ "status":0, "QTime":0, "params":{ "q":"box:8864fa1d51ea1d5a7b12000096a1aaf8 AND user:fa) doveadm(fail2ban@mydomain.com): Error: Mailbox INBOX: Status lookup failed: Internal error occurred. Refer to server log for more information. [2017-11-30 13:05:24]
My system is CentOS 7, Dovecot 3.2.4 (installed from source with solr support), Java OpenJDK-1.8.0, Solr v7.1.0.
The link you posted is for solr 6.5.0 and you are using solr 7.1.0 and solr needs to be configured to accept XML queries instead of json as the default starting with version 7. Search the list archives there have been postings regarding solr 7 and dovecot in the past.
Could some one help me on this please?
Here is detailed debug out put:
[root@mail ~]# doveadm -vD index -u fail2ban@mydomain.com Inbox Debug: Loading modules from directory: /usr/lib64/dovecot Debug: Module loaded: /usr/lib64/dovecot/lib10_quota_plugin.so Debug: Module loaded: /usr/lib64/dovecot/lib20_fts_plugin.so Debug: Module loaded: /usr/lib64/dovecot/lib21_fts_solr_plugin.so Debug: Loading modules from directory: /usr/lib64/dovecot/doveadm Debug: Skipping module doveadm_acl_plugin, because dlopen() failed: /usr/lib64/dovecot/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol: acl_user_module (this is usually intentional, so just ignore this message) Debug: Skipping module doveadm_expire_plugin, because dlopen() failed: /usr/lib64/dovecot/doveadm/lib10_doveadm_expire_plugin.so: undefined symbol: expire_set_deinit (this is usually intentional, so just ignore this message) Debug: Module loaded: /usr/lib64/dovecot/doveadm/lib10_doveadm_quota_plugin.so Debug: Module loaded: /usr/lib64/dovecot/doveadm/lib10_doveadm_sieve_plugin.so Debug: Module loaded: /usr/lib64/dovecot/doveadm/lib20_doveadm_fts_plugin.so Debug: Skipping module doveadm_mail_crypt_plugin, because dlopen() failed: /usr/lib64/dovecot/doveadm/libdoveadm_mail_crypt_plugin.so: undefined symbol: mail_crypt_box_get_pvt_digests (this is usually intentional, so just ignore this message) doveadm(fail2ban@mydomain.com): Debug: Added userdb setting: mail=maildir:/home/vmail/mydomain.com/fail2ban/ doveadm(fail2ban@mydomain.com): Debug: Added userdb setting: plugin/quota_rule=*:bytes=0 doveadm(fail2ban@mydomain.com): Debug: Effective uid=5000, gid=5000, home=/home/vmail/mydomain.com/fail2ban/ doveadm(fail2ban@mydomain.com): Debug: Quota root: name=User quota backend=maildir args= doveadm(fail2ban@mydomain.com): Debug: Quota rule: root=User quota mailbox=* bytes=0 messages=0 doveadm(fail2ban@mydomain.com): Debug: Quota rule: root=User quota mailbox=Trash bytes=+104857600 messages=0 doveadm(fail2ban@mydomain.com): Debug: Quota warning: bytes=0 (90%) messages=0 reverse=no command=quota-warning 90 fail2ban@mydomain.com mydomain.com doveadm(fail2ban@mydomain.com): Debug: Quota warning: bytes=0 (80%) messages=0 reverse=no command=quota-warning 80 fail2ban@mydomain.com mydomain.com doveadm(fail2ban@mydomain.com): Debug: Quota grace: root=User quota bytes=0 (10%) doveadm(fail2ban@mydomain.com): Debug: Namespace inbox: type=private, prefix=, sep=, inbox=yes, hidden=no, list=yes, subscriptions=yes location=maildir:/home/vmail/mydomain.com/fail2ban/ doveadm(fail2ban@mydomain.com): Debug: maildir++: root=/home/vmail/mydomain.com/fail2ban, index=, indexpvt=, control=, inbox=/home/vmail/mydomain.com/fail2ban, alt= doveadm(fail2ban@mydomain.com): Debug: quota: quota_over_flag check: quota_over_script unset - skipping doveadm(fail2ban@mydomain.com): Debug: INBOX: Mailbox opened because: index doveadm(fail2ban@mydomain.com): Debug: http-client: host 127.0.0.1: Host created doveadm(fail2ban@mydomain.com): Debug: http-client: peer 127.0.0.1:8983: Peer created doveadm(fail2ban@mydomain.com): Debug: http-client: queue http://127.0.0.1:8983: Setting up connection to 127.0.0.1:8983 (1 requests pending) doveadm(fail2ban@mydomain.com): Debug: http-client: peer 127.0.0.1:8983: Linked queue http://127.0.0.1:8983 (1 queues linked) doveadm(fail2ban@mydomain.com): Debug: http-client: queue http://127.0.0.1:8983: Started new connection to 127.0.0.1:8983 doveadm(fail2ban@mydomain.com): Debug: http-client: request [Req1: GET http://127.0.0.1:8983/solr/dovecot/select?fl=uid&rows=1&sort=uid+desc&q=box:8864fa1d51ea1d5a7b12000096a1aaf8+AND+user:fail2ban@mydomain.com]:
Submitted doveadm(fail2ban@mydomain.com): Debug: http-client: Waiting for 1 requests to finish doveadm(fail2ban@mydomain.com): Debug: http-client: peer 127.0.0.1:8983: Creating 1 new connections to handle requests (already 0 usable, connecting to 0, closing 0) doveadm(fail2ban@mydomain.com): Debug: http-client: peer 127.0.0.1:8983: Making new connection 1 of 1 doveadm(fail2ban@mydomain.com): Debug: http-client: conn 127.0.0.1:8983
0: HTTP connection created (1 parallel connections exist) doveadm(fail2ban@mydomain.com): Debug: http-client: conn 127.0.0.1:8983
doveadm(fail2ban@mydomain.com): Debug: http-client: conn 127.0.0.1:8983
0: Ready for requests doveadm(fail2ban@mydomain.com): Debug: http-client: peer 127.0.0.1:8983: Successfully connected (connections=1) doveadm(fail2ban@mydomain.com): Debug: http-client: peer 127.0.0.1:8983: Using 1 idle connections to handle 1 requests (1 total connections ready) doveadm(fail2ban@mydomain.com): Debug: http-client: queue http://127.0.0.1:8983: Connection to peer 127.0.0.1:8983 claimed request [Req1: GET http://127.0.0.1:8983/solr/dovecot/select?fl=uid&rows=1&sort=uid+desc&q=box:8864fa1d51ea1d5a7b12000096a1aaf8+AND+user:fail2ban@mydomain.com]
doveadm(fail2ban@mydomain.com): Debug: http-client: conn 127.0.0.1:8983
0: Claimed request [Req1: GET http://127.0.0.1:8983/solr/dovecot/select?fl=uid&rows=1&sort=uid+desc&q=box:8864fa1d51ea1d5a7b12000096a1aaf8+AND+user:fail2ban@mydomain.com] doveadm(fail2ban@mydomain.com): Debug: http-client: request [Req1: GET http://127.0.0.1:8983/solr/dovecot/select?fl=uid&rows=1&sort=uid+desc&q=box:8864fa1d51ea1d5a7b12000096a1aaf8+AND+user:fail2ban@mydomain.com]:
Sent header doveadm(fail2ban@mydomain.com): Debug: http-client: peer 127.0.0.1:8983: No more requests to service for this peer (1 connections exist) doveadm(fail2ban@mydomain.com): Debug: http-client: conn 127.0.0.1:8983
0: Got 200 response for request [Req1: GET http://127.0.0.1:8983/solr/dovecot/select?fl=uid&rows=1&sort=uid+desc&q=box:8864fa1d51ea1d5a7b12000096a1aaf8+AND+user:fail2ban@mydomain.com]
(took 3 ms + 2 ms in queue) doveadm(fail2ban@mydomain.com): Error: fts_solr: Invalid XML input at 1:0: not well-formed (invalid token) (near: { "responseHeader":{ "status":0, "QTime":1, "params":{ "q":"box:8864fa1d51ea1d5a7b12000096a1aaf8 AND user:fa) doveadm(fail2ban@mydomain.com): Debug: http-client: conn 127.0.0.1:8983
0: Response payload stream destroyed (0 ms after initial response) doveadm(fail2ban@mydomain.com): Debug: http-client: request [Req1: GET http://127.0.0.1:8983/solr/dovecot/select?fl=uid&rows=1&sort=uid+desc&q=box:8864fa1d51ea1d5a7b12000096a1aaf8+AND+user:fail2ban@mydomain.com]:
Finished doveadm(fail2ban@mydomain.com): Debug: http-client: queue http://127.0.0.1:8983: Dropping request [Req1: GET http://127.0.0.1:8983/solr/dovecot/select?fl=uid&rows=1&sort=uid+desc&q=box:8864fa1d51ea1d5a7b12000096a1aaf8+AND+user:fail2ban@mydomain.com] doveadm(fail2ban@mydomain.com): Debug: http-client: host 127.0.0.1: Host is idle (timeout = 100 msecs) doveadm(fail2ban@mydomain.com): Debug: http-client: request [Req1: GET http://127.0.0.1:8983/solr/dovecot/select?fl=uid&rows=1&sort=uid+desc&q=box:8864fa1d51ea1d5a7b12000096a1aaf8+AND+user:fail2ban@mydomain.com]:
Free (requests left=1) doveadm(fail2ban@mydomain.com): Debug: http-client: All requests finished doveadm(fail2ban@mydomain.com): Error: Mailbox INBOX: Status lookup failed: Internal error occurred. Refer to server log for more information. [2017-11-30 13:13:57] doveadm(fail2ban@mydomain.com): Debug: http-client: peer 127.0.0.1:8983: Peer close doveadm(fail2ban@mydomain.com): Debug: http-client: peer 127.0.0.1:8983: Peer disconnect doveadm(fail2ban@mydomain.com): Debug: http-client: conn 127.0.0.1:8983
0: Peer closed doveadm(fail2ban@mydomain.com): Debug: http-client: conn 127.0.0.1:8983
0: Connection disconnect doveadm(fail2ban@mydomain.com): Debug: http-client: conn 127.0.0.1:8983
0: Connection destroy doveadm(fail2ban@mydomain.com): Debug: http-client: peer 127.0.0.1:8983: Peer destroy doveadm(fail2ban@mydomain.com): Debug: http-client: host 127.0.0.1: Host destroy doveadm(fail2ban@mydomain.com): Debug: http-client: queue http://127.0.0.1:8983: Destroy
-- Christian Kivalo
Thanks a lot for the help.
You are right about the JSON as the default in solr v7.1.0. I now switched it back to XML and now it works.
# doveadm -v index -u fail2ban@mydomain.com Inbox doveadm(fail2ban@mydomain.com): Info: INBOX: Caching mails seq=1..388 388/388
Gao
On 2017-11-30 02:00 PM, Christian Kivalo wrote:
solr 7 and dovecot
participants (2)
-
Christian Kivalo
-
Gao