[Dovecot] panic in dovecot when using fts with solr
Hello All,
I have a panic error in dovecot when I try to use the fts search with Solr.
The version of dovecot I use is version 2.1
I have used a telnet session to search, but I don't know if this make a difference:
telnet localhost imap Trying 127.0.0.1... Connected to localhost. Escape character is '^]'.
- OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE STARTTLS AUTH=PLAIN AUTH=LOGIN] Dovecot ready. 1 login andre-rodier XXXXXXXXXX 1 OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS MULTIAPPEND UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS SPECIAL-USE SEARCH=FUZZY QUOTA ACL RIGHTS=texk] Logged in 2 select Inbox
- FLAGS (\Answered \Flagged \Deleted \Seen \Draft NonJunk $labels_4f9cce7caf4df $Forwarded $MDNSent Junk $labels_50f53a5862fb9 $label2 $label1 $label4 $label3 announce announces $ulabels_50f548e03429e feedback)
- OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft NonJunk $labels_4f9cce7caf4df $Forwarded $MDNSent Junk $labels_50f53a5862fb9 $label2 $label1 $label4 $label3 announce announces $ulabels_50f548e03429e feedback \*)] Flags permitted.
- 2158 EXISTS
- 0 RECENT
- OK [UIDVALIDITY 1354776694] UIDs valid
- OK [UIDNEXT 2568] Predicted next UID
- OK [HIGHESTMODSEQ 9739] Highest 2 OK [READ-WRITE] Select completed. 3 search text Valentine
- OK Searched 47% of the mailbox, ETA 0:10
- OK Searched 79% of the mailbox, ETA 0:05
- SEARCH 1120 1238 1279 1975 1990 1991 1992 1998 1999 2000 2001 2002 2058 2106 2107 2110 2151 3 OK Search completed (32.210 secs).
Mar 2 22:48:02 lapetus dovecot: indexer-worker(andre rodier@indiefield.co.uk): Panic: file solr-connection.c: line 545 (solr_connection_post_more): assertion failed: (maxfd >= 0) Mar 2 22:48:02 lapetus dovecot: indexer-worker(andre rodier@indiefield.co.uk): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x4414a) [0x7f5f4237214a] -> /usr/lib/dovecot/libdovecot.so.0(+0x4418e) [0x7f5f4237218e] -> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7f5f4234982a] -> /usr/lib/dovecot/modules/lib21_fts_solr_plugin.so(solr_connection_post_more+0x249) [0x7f5f402d48d9] -> /usr/lib/dovecot/modules/lib21_fts_solr_plugin.so(+0x4ac7) [0x7f5f402d1ac7] -> /usr/lib/dovecot/modules/lib20_fts_plugin.so(fts_build_mail+0x339) [0x7f5f408f1f19] -> /usr/lib/dovecot/modules/lib20_fts_plugin.so(+0xc6e0) [0x7f5f408f66e0] -> dovecot/indexer-worker(+0x28d4) [0x7f5f42ae88d4] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x36) [0x7f5f4237f016] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0xa7) [0x7f5f4237fcc7] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x28) [0x7f5f4237ea28] -> /usr/lib/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7f5f4236b483] -> dovecot/indexer-worker(main+0xfe) [0x7f5f42ae834e] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xfd) [0x7f5f41fc2ead] -> dovecot/indexer-worker(+0x23f5) [0x7f5f42ae83f5] Mar 2 22:48:02 lapetus dovecot: indexer: Error: Indexer worker disconnected, discarding 1 requests for andre rodier@indiefield.co.uk Mar 2 22:48:02 lapetus dovecot: imap(andre rodier@indiefield.co.uk): Error: indexer failed to index mailbox INBOX Mar 2 22:48:02 lapetus dovecot: indexer-worker(andre rodier@indiefield.co.uk): Fatal: master: service(indexer-worker): child 19644 killed with signal 6 (core dumps disabled)
Regards, Andre
Hello again,
Sorry for the duplicate post earlier.
My dovecot version is 2.1.7-7 (included in Debian Wheezy)
I don't think that addinf doveconf output would be useful, but I can add it if you are interested.
Kind regards, André Rodier
On 02/03/13 22:57, Andre Rodier wrote:
Hello All,
I have a panic error in dovecot when I try to use the fts search with Solr.
The version of dovecot I use is version 2.1
I have used a telnet session to search, but I don't know if this make a difference:
telnet localhost imap Trying 127.0.0.1... Connected to localhost. Escape character is '^]'.
- OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE STARTTLS AUTH=PLAIN AUTH=LOGIN] Dovecot ready. 1 login andre-rodier XXXXXXXXXX 1 OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS MULTIAPPEND UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS SPECIAL-USE SEARCH=FUZZY QUOTA ACL RIGHTS=texk] Logged in 2 select Inbox
- FLAGS (\Answered \Flagged \Deleted \Seen \Draft NonJunk $labels_4f9cce7caf4df $Forwarded $MDNSent Junk $labels_50f53a5862fb9 $label2 $label1 $label4 $label3 announce announces $ulabels_50f548e03429e feedback)
- OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft NonJunk $labels_4f9cce7caf4df $Forwarded $MDNSent Junk $labels_50f53a5862fb9 $label2 $label1 $label4 $label3 announce announces $ulabels_50f548e03429e feedback \*)] Flags permitted.
- 2158 EXISTS
- 0 RECENT
- OK [UIDVALIDITY 1354776694] UIDs valid
- OK [UIDNEXT 2568] Predicted next UID
- OK [HIGHESTMODSEQ 9739] Highest 2 OK [READ-WRITE] Select completed. 3 search text Valentine
- OK Searched 47% of the mailbox, ETA 0:10
- OK Searched 79% of the mailbox, ETA 0:05
- SEARCH 1120 1238 1279 1975 1990 1991 1992 1998 1999 2000 2001 2002 2058 2106 2107 2110 2151 3 OK Search completed (32.210 secs).
Mar 2 22:48:02 lapetus dovecot: indexer-worker(andre rodier@indiefield.co.uk): Panic: file solr-connection.c: line 545 (solr_connection_post_more): assertion failed: (maxfd >= 0) Mar 2 22:48:02 lapetus dovecot: indexer-worker(andre rodier@indiefield.co.uk): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x4414a) [0x7f5f4237214a] -> /usr/lib/dovecot/libdovecot.so.0(+0x4418e) [0x7f5f4237218e] -> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7f5f4234982a] -> /usr/lib/dovecot/modules/lib21_fts_solr_plugin.so(solr_connection_post_more+0x249) [0x7f5f402d48d9] -> /usr/lib/dovecot/modules/lib21_fts_solr_plugin.so(+0x4ac7) [0x7f5f402d1ac7] -> /usr/lib/dovecot/modules/lib20_fts_plugin.so(fts_build_mail+0x339) [0x7f5f408f1f19] -> /usr/lib/dovecot/modules/lib20_fts_plugin.so(+0xc6e0) [0x7f5f408f66e0] -> dovecot/indexer-worker(+0x28d4) [0x7f5f42ae88d4] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x36) [0x7f5f4237f016] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0xa7) [0x7f5f4237fcc7] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x28) [0x7f5f4237ea28] -> /usr/lib/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7f5f4236b483] -> dovecot/indexer-worker(main+0xfe) [0x7f5f42ae834e] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xfd) [0x7f5f41fc2ead] -> dovecot/indexer-worker(+0x23f5) [0x7f5f42ae83f5] Mar 2 22:48:02 lapetus dovecot: indexer: Error: Indexer worker disconnected, discarding 1 requests for andre rodier@indiefield.co.uk Mar 2 22:48:02 lapetus dovecot: imap(andre rodier@indiefield.co.uk): Error: indexer failed to index mailbox INBOX Mar 2 22:48:02 lapetus dovecot: indexer-worker(andre rodier@indiefield.co.uk): Fatal: master: service(indexer-worker): child 19644 killed with signal 6 (core dumps disabled)
Regards, Andre
This is my doveconf output. Please, note that since the panic, I have switched to fts with squat, but it is very limited for large mailboxes.
Any idea how to fix this?
# 2.1.7: /etc/dovecot/dovecot.conf # OS: Linux 3.2.0-4-amd64 x86_64 Debian 7.0 auth_default_realm = indiefield.co.uk auth_master_user_separator = : auth_mechanisms = plain login auth_username_chars = abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890 .-_@ auth_username_translation = "- _ " auth_verbose = yes disable_plaintext_auth = no dotlock_use_excl = no hostname = smtp.indiefield.org.uk lda_mailbox_autocreate = yes lda_mailbox_autosubscribe = yes lock_method = dotlock mail_access_groups = vmail mail_fsync = always mail_gid = vmail mail_location = maildir:~/Mails/Maildir/:INDEX=~/Mails/Indexes/ mail_plugins = autocreate quota acl virtual listescape fts fts_squat mail_privileged_group = vmail mail_uid = vmail maildir_copy_with_hardlinks = no managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave mmap_disable = yes namespace { hidden = no inbox = no list = yes location = maildir:/var/vmail/%d/Public/Mails/Maildir/:INDEX=~/Mails/PublicIndexes/ prefix = Public/ separator = / subscriptions = yes type = public } namespace { list = children location = maildir:%%h/Mails/Maildir/:INDEX=~/Mails/SharedIndexes/%%u prefix = Shared/%%d/%%n/ separator = / subscriptions = no type = shared } namespace inbox { hidden = no inbox = yes list = yes location = maildir:~/Mails/Maildir/:INDEX=~/Mails/Indexes/ mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = separator = / subscriptions = yes type = private } passdb { args = /etc/dovecot/dovecot-ldap.conf.ext driver = ldap } plugin { acl = vfile:/etc/dovecot/global-acls:cache_secs=300 acl_anyone = allow acl_shared_dict = file:/var/vmail/indiefield.co.uk/shared-mailboxes autocreate = Trash autocreate2 = Drafts autocreate3 = Sent autocreate4 = Junk autocreate5 = Archives autosubscribe = Trash autosubscribe2 = Drafts autosubscribe3 = Sent autosubscribe4 = Junk autosubscribe5 = Archives fts = squat fts_squat = partial=4 full=10 listescape_char = %% quota = maildir:User quota quota_warning = storage=95%% quota-warning 95 %u sieve = ~/Mails/.dovecot.sieve sieve_after = %h/Mails/Sieve/After/ sieve_before = /etc/dovecot/sieve-default/before/ sieve_dir = ~/Mails/Sieve sieve_max_script_size = 4M } protocols = " imap lmtp sieve pop3" service auth-worker { user = vmail } service auth { unix_listener /var/spool/postfix/private/auth { group = vmail mode = 0666 user = postfix } unix_listener auth-userdb { group = vmail mode = 0666 user = postfix } user = vmail } service dict { unix_listener dict { group = vmail mode = 0600 user = vmail } } service director { fifo_listener login/proxy-notify { group = postfix mode = 0666 user = postfix } unix_listener director-userdb { group = postfix mode = 0600 user = postfix } unix_listener login/director { group = postfix mode = 0666 user = postfix } } service lmtp { group = vmail inet_listener lmtp { port = 24 } user = vmail } service managesieve-login { inet_listener sieve { port = 4190 } } service quota-warning { executable = script /etc/dovecot/scripts/quota-warning.sh unix_listener quota-warning { user = vmail } user = dovecot } ssl_cert =
Ok,
I found the issue.
The solr schema xml file was wrong, and I did not check the error logs correctly. So, the error was in solr:
04-Mar-2013 15:15:30 org.apache.solr.common.SolrException log SEVERE: org.apache.solr.common.SolrException: ERROR:unknown field 'uid' at org.apache.solr.update.DocumentBuilder.toDocument(DocumentBuilder.java:289) at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:60) at org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:139) at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:69) at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:54) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1317) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:943) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:843) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
04-Mar-2013 15:15:30 org.apache.solr.core.SolrCore execute INFO: [] webapp=/solr path=/update params={} status=400 QTime=70
However, IMHO, I don't think that dovecot handle the errors correctly. It should return an error message.
Kind regards, André Rodier
On 4 March 2013 10:01, André Rodier andre.rodier@gmail.com wrote:
This is my doveconf output. Please, note that since the panic, I have switched to fts with squat, but it is very limited for large mailboxes.
Any idea how to fix this?
# 2.1.7: /etc/dovecot/dovecot.conf # OS: Linux 3.2.0-4-amd64 x86_64 Debian 7.0 auth_default_realm = indiefield.co.uk auth_master_user_separator = : auth_mechanisms = plain login auth_username_chars = abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890 .-_@ auth_username_translation = "- _ " auth_verbose = yes disable_plaintext_auth = no dotlock_use_excl = no hostname = smtp.indiefield.org.uk lda_mailbox_autocreate = yes lda_mailbox_autosubscribe = yes lock_method = dotlock mail_access_groups = vmail mail_fsync = always mail_gid = vmail mail_location = maildir:~/Mails/Maildir/:INDEX=~/Mails/Indexes/ mail_plugins = autocreate quota acl virtual listescape fts fts_squat mail_privileged_group = vmail mail_uid = vmail maildir_copy_with_hardlinks = no managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave mmap_disable = yes namespace { hidden = no inbox = no list = yes location = maildir:/var/vmail/%d/Public/Mails/Maildir/:INDEX=~/Mails/PublicIndexes/ prefix = Public/ separator = / subscriptions = yes type = public } namespace { list = children location = maildir:%%h/Mails/Maildir/:INDEX=~/Mails/SharedIndexes/%%u prefix = Shared/%%d/%%n/ separator = / subscriptions = no type = shared } namespace inbox { hidden = no inbox = yes list = yes location = maildir:~/Mails/Maildir/:INDEX=~/Mails/Indexes/ mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = separator = / subscriptions = yes type = private } passdb { args = /etc/dovecot/dovecot-ldap.conf.ext driver = ldap } plugin { acl = vfile:/etc/dovecot/global-acls:cache_secs=300 acl_anyone = allow acl_shared_dict = file:/var/vmail/indiefield.co.uk/shared-mailboxes autocreate = Trash autocreate2 = Drafts autocreate3 = Sent autocreate4 = Junk autocreate5 = Archives autosubscribe = Trash autosubscribe2 = Drafts autosubscribe3 = Sent autosubscribe4 = Junk autosubscribe5 = Archives fts = squat fts_squat = partial=4 full=10 listescape_char = %% quota = maildir:User quota quota_warning = storage=95%% quota-warning 95 %u sieve = ~/Mails/.dovecot.sieve sieve_after = %h/Mails/Sieve/After/ sieve_before = /etc/dovecot/sieve-default/before/ sieve_dir = ~/Mails/Sieve sieve_max_script_size = 4M } protocols = " imap lmtp sieve pop3" service auth-worker { user = vmail } service auth { unix_listener /var/spool/postfix/private/auth { group = vmail mode = 0666 user = postfix } unix_listener auth-userdb { group = vmail mode = 0666 user = postfix } user = vmail } service dict { unix_listener dict { group = vmail mode = 0600 user = vmail } } service director { fifo_listener login/proxy-notify { group = postfix mode = 0666 user = postfix } unix_listener director-userdb { group = postfix mode = 0600 user = postfix } unix_listener login/director { group = postfix mode = 0666 user = postfix } } service lmtp { group = vmail inet_listener lmtp { port = 24 } user = vmail } service managesieve-login { inet_listener sieve { port = 4190 } } service quota-warning { executable = script /etc/dovecot/scripts/quota-warning.sh unix_listener quota-warning { user = vmail } user = dovecot } ssl_cert =
-- André Rodier
participants (2)
-
Andre Rodier
-
André Rodier