fts_solr not working

Jeff Gamsby jeffgamsby at merlock.com
Sun Jun 26 18:56:15 UTC 2016


On 2016-06-23 11:24, aki.tuomi at dovecot.fi wrote:
>> On June 23, 2016 at 9:06 PM Jeff Gamsby <jeffgamsby at merlock.com> 
>> wrote:
>> 
>> 
>> I am running ISPConfig 3 on Debian and have managed to install the
>> dovecot-solr and dovecot-fts plugins.
>> 
>> I have solr running undet tomcat at http://localhost:8880 but the
>> indexing is not working.
>> 
>> I am using Dovecot 2.17
>> 
>> I do not understand namespaces and why fts_solr needs them, I just 
>> want
>> to index the entire users Maildir.
>> 
>> I am trying to index a users mailbox but am getting the following 
>> error:
>> 
>> (changed user name)
>> doveadm fts rescan -u user at user.com inbox
>> doveadm(user at user.com): Error: Namespace prefix not found: inbox
>> 
>> running that in debig mode gives:
>> 
>> doveadm(root): Debug: Loading modules from directory:
>> /usr/lib/dovecot/modules
>> doveadm(root): Debug: Module loaded:
>> /usr/lib/dovecot/modules/lib20_fts_plugin.so
>> doveadm(root): Debug: Module loaded:
>> /usr/lib/dovecot/modules/lib21_fts_solr_plugin.so
>> doveadm(root): Debug: Loading modules from directory:
>> /usr/lib/dovecot/modules/doveadm
>> doveadm(root): Debug: Skipping module doveadm_acl_plugin, because
>> dlopen() failed:
>> /usr/lib/dovecot/modules/doveadm/lib10_doveadm_acl_plugin.so: 
>> undefined
>> symbol: acl_user_module (this is usually intentional, so just ignore
>> this message)
>> doveadm(root): Debug: Skipping module doveadm_expire_plugin, because
>> dlopen() failed:
>> /usr/lib/dovecot/modules/doveadm/lib10_doveadm_expire_plugin.so:
>> undefined symbol: expire_set_deinit (this is usually intentional, so
>> just ignore this message)
>> doveadm(root): Debug: Skipping module doveadm_quota_plugin, because
>> dlopen() failed:
>> /usr/lib/dovecot/modules/doveadm/lib10_doveadm_quota_plugin.so:
>> undefined symbol: quota_user_module (this is usually intentional, so
>> just ignore this message)
>> doveadm(root): Debug: Skipping module doveadm_zlib_plugin, because
>> dlopen() failed:
>> /usr/lib/dovecot/modules/doveadm/lib10_doveadm_zlib_plugin.so: 
>> undefined
>> symbol: i_stream_create_deflate (this is usually intentional, so just
>> ignore this message)
>> doveadm(root): Debug: Module loaded:
>> /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_plugin.so
>> doveadm(user at user.com): Debug: Added userdb setting:
>> mail=maildir:/var/vmail/user.com/user/Maildir
>> doveadm(user at user.com): Debug: Added userdb setting:
>> plugin/quota_rule=*:storage=0B
>> doveadm(user at user.com): Debug: Added userdb setting:
>> plugin/sieve=/var/vmail/user.com/user/.sieve
>> doveadm(user at user.com): Debug: Effective uid=5000, gid=5000,
>> home=/var/vmail/user.com/user
>> doveadm(user at user.com): Debug: Namespace inbox: type=private, prefix=,
>> sep=, inbox=yes, hidden=no, list=yes, subscriptions=yes
>> location=maildir:/var/vmail/user.com/user/Maildir
>> doveadm(user at user.com): Debug: maildir++:
>> root=/var/vmail/user.com/user/Maildir, index=, control=,
>> inbox=/var/vmail/user.com/user/Maildir, alt=
>> doveadm(user at user.com): Debug: fts: No fts setting - plugin disabled
>> doveadm(user at user.com): Error: Namespace prefix not found: inbox
>> 
>> in conf.d/10-mail.conf I have this namespace defined:
>> 
>> namespace inbox {
>>    # Namespace type: private, shared or public
>>    #type = private
>> 
>>    # Hierarchy separator to use. You should use the same separator for
>> all
>>    # namespaces or some clients get confused. '/' is usually a good 
>> one.
>>    # The default however depends on the underlying mail storage 
>> format.
>>    #separator =
>> 
>>    # Prefix required to access this namespace. This needs to be 
>> different
>> for
>>    # all namespaces. For example "Public/".
>>    #prefix =
>> 
>>    # Physical location of the mailbox. This is in same format as
>>    # mail_location, which is also the default for it.
>>    #location =
>> 
>>    # There can be only one INBOX, and this setting defines which
>> namespace
>>    # has it.
>>    inbox = yes
>> 
>>    # If namespace is hidden, it's not advertised to clients via 
>> NAMESPACE
>>    # extension. You'll most likely also want to set list=no. This is
>> mostly
>>    # useful when converting from another server with different 
>> namespaces
>> which
>>    # you want to deprecate but still keep working. For example you can
>> create
>>    # hidden namespaces with prefixes "~/mail/", "~%u/mail/" and 
>> "mail/".
>>    #hidden = no
>> protocol imap {
>>    plugin {
>>      fts = solr
>>      fts_solr = break-imap-search url=http://localhost:8880/solr/
>>    }
>> }
>> protocol pop3 {
>>    plugin {
>>      fts = solr
>>      fts_solr = break-imap-search url=http://localhost:8880/solr/
>>    }
>> 
>>    # Show the mailboxes under this namespace with LIST command. This
>> makes the
>>    # namespace visible for clients that don't support NAMESPACE
>> extension.
>>    # "children" value lists child mailboxes, but hides the namespace
>> prefix.
>>    #list = yes
>> 
>>    # Namespace handles its own subscriptions. If set to "no", the 
>> parent
>>    # namespace handles them (empty prefix should always have this as
>> "yes")
>>    #subscriptions = yes
>> }
>> 
>> 
>> I had to put this in dovecot.conf in order for the plugin to be 
>> enabled:
>> 
>> mail_plugins = fts fts_solr
>> 
>> solr is reachable at localhost:8880/solr and appears to be working.
>> 
>> Please help, any suggestions are welcome
>> 
>> Thanks
> 
> Can you please send doveconf -n?
> ---
> Aki Tuomi


I managed to get fts_slor working and now I can index mailboxes, but I 
am getting a solr error and cannot use the indexes that were created

I get:

Error: fts_solr: Lookup failed: Internal Server Error


dovecot -n

# 2.2.13: /etc/dovecot/dovecot.conf
# OS: Linux 3.2.0-4-amd64 x86_64 Debian 7.1
auth_mechanisms = plain login
default_vsz_limit = 2 G
disable_plaintext_auth = no
lda_mailbox_autocreate = yes
lda_mailbox_autosubscribe = yes
listen = *,[::]
log_timestamp = "%Y-%m-%d %H:%M:%S "
mail_plugins = fts fts_solr
mail_privileged_group = vmail
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope 
encoded-character vacation subaddress comparator-i;ascii-numeric 
relational regex imap4flags copy include variables body enotify 
environment mailbox date ihave
namespace inbox {
   inbox = yes
   location =
   prefix =
}
passdb {
   args = /etc/dovecot/dovecot-sql.conf
   driver = sql
}
plugin {
   fts = solr
   fts_autoindex = yes
   fts_solr = break-imap-search url=http://localhost:8880/solr/
   quota = dict:user::file:/var/vmail/%d/%n/.quotausage
   sieve = ~/.dovecot.sieve
   sieve_after = /etc/sieve/after
   sieve_before = /etc/sieve/before
   sieve_default = /var/vmail/sieve/default.sieve
   sieve_dir = ~/sieve
}
protocols = imap pop3
service auth {
   unix_listener /var/spool/postfix/private/auth {
     group = postfix
     mode = 0660
     user = postfix
   }
   unix_listener auth-master {
     group = vmail
     mode = 0660
     user = vmail
   }
   unix_listener auth-userdb {
     group = vmail
     mode = 0600
     user = vmail
   }
   user = root
}
service imap-login {
   client_limit = 1000
   process_limit = 500
}
service imap {
   vsz_limit = 2 G
}
ssl_cert = </etc/postfix/smtpd.cert
ssl_key = </etc/postfix/smtpd.key
userdb {
   args = /etc/dovecot/dovecot-sql.conf
   driver = sql
}
protocol pop3 {
   pop3_uidl_format = %08Xu%08Xv
}
protocol lda {
   auth_socket_path = /var/run/dovecot/auth-master
   info_log_path = /var/log/dovecot-lda.log
   log_path = /var/log/dovecot-lda-errors.log
   mail_plugins = fts fts_solr autocreate sieve quota
   postmaster_address = postmaster at user.com
}



And this shows that indexes are correct:

doveadm -Dv index -u jeff at user.com testing

doveadm(jeff at user.com): Info: testing: Cache is already up to date


using solr-schema.xml from dovecot-2.2.13 sources in 
/etc/solr/conf/schema.xml

dovecot-sql.conf

# http://wiki.dovecot.org/AuthDatabase/SQL
#
# CREATE TABLE users (
#     userid VARCHAR(128) NOT NULL,
#     password VARCHAR(64) NOT NULL,
#     home VARCHAR(255) NOT NULL,
#     uid INTEGER NOT NULL,
#     gid INTEGER NOT NULL,
#     active CHAR(1) DEFAULT 'Y' NOT NULL
# );

driver = mysql
connect = host=localhost dbname=dbispconfig user=ispconfig 
password=xxxxxxxxxxxxxxxxxxxxx
default_pass_scheme = CRYPT

password_query = SELECT password FROM mail_user WHERE (login = '%u' OR 
email = '%u') AND `disable%Ls` = 'n' AND server_id = '1'
user_query = SELECT email as user, maildir as home, CONCAT('maildir:', 
maildir, '/Maildir') as mail, uid, gid, CONCAT('*:storage=', quota, 'B') 
AS quota_rule, CONCAT(maildir, '/.sieve') as sieve FROM mail_user WHERE 
(login = '%u' OR email = '%u') AND `disable%Ls` = 'n' AND server_id = 
'1'

# The iterate_query is required for the doveadm command only and works 
only on dovecot 2 servers.
# Do not enable it on Dovecot 1.x servers
iterate_query = SELECT email as user FROM mail_user


More information about the dovecot mailing list