[Dovecot] fts solr : out of memory

tonio at starbridge.org tonio at starbridge.org
Tue Jul 31 22:46:54 EEST 2012 

Le 31/07/2012 21:04, Timo Sirainen a écrit :
> On 29.7.2012, at 2.20, tonio at starbridge.org wrote:
>
>> it seems ok now with memory problem, but an other error shows up:
>>
>> doveadm -v index -u clxx at spamguard.fr INBOX
>> doveadm(clxx at spamguard.fr): Info: INBOX: Caching mails seq=1..59363
>> 21500/59363doveadm(clxx at spamguard.fr): Error: fts_solr: Invalid XML
>> input at line 1: mismatched tag
> No idea. You can reproduce this? What does it log with this patch? http://hg.dovecot.org/dovecot-2.1/rev/817b69b2b21f

It happens every time on the same mailboxes (very few) around the same
uid number (I think I can find the exact uid with strace and send the
email message to you if it helps)

catalina.out show this at this time:

INFO: {} 0 1
31 juil. 2012 21:19:56 org.apache.solr.common.SolrException log
GRAVE: org.apache.solr.common.SolrException: Illegal character
((CTRL-CHAR, code 4))
 at [row,col {unknown-source}]: [533,500]
        at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:81)
        at
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:58)
        at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:1376)
        at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:365)
        at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:260)
        at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
        at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
        at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
        at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
        at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:602)
        at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
        at java.lang.Thread.run(Unknown Source)
Caused by: com.ctc.wstx.exc.WstxUnexpectedCharException: Illegal
character ((CTRL-CHAR, code 4))
 at [row,col {unknown-source}]: [533,500]
        at
com.ctc.wstx.sr.StreamScanner.throwInvalidSpace(StreamScanner.java:675)
        at
com.ctc.wstx.sr.BasicStreamReader.readTextSecondary(BasicStreamReader.java:4668)
        at
com.ctc.wstx.sr.BasicStreamReader.readCoalescedText(BasicStreamReader.java:4126)
        at
com.ctc.wstx.sr.BasicStreamReader.finishToken(BasicStreamReader.java:3701)
        at
com.ctc.wstx.sr.BasicStreamReader.safeFinishToken(BasicStreamReader.java:3649)
        at
com.ctc.wstx.sr.BasicStreamReader.getText(BasicStreamReader.java:809)
        at org.apache.solr.handler.XMLLoader.readDoc(XMLLoader.java:315)
        at
org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:156)
        at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:79)
        ... 17 more

31 juil. 2012 21:19:56 org.apache.solr.core.SolrCore execute
INFO: [] webapp=/solr path=/update params={} status=400 QTime=1

After a quick google search , it seems related to invalid Control
Character sent to SOLR.



I've applied your last patch and the message is now:
Error: fts_solr: Invalid XML input at 4:113: mismatched tag (near:
<html><head><title>Apache Tomcat/6.0.35 - Rapport
d'erreur</title><style><!--H1
{font-family:Tahoma,Arial,sans-serif;color:white)
doveadm(xx at xx): Panic: file solr-connection.c: line 547
(solr_connection_post_more): assertion failed: (maxfd >= 0)
doveadm(xx at xx): Error: Raw backtrace:
/usr/lib/dovecot/libdovecot.so.0(+0x43d9a) [0x7fadd7e1dd9a] ->
/usr/lib/dovecot/libdovecot.so.0(default_fatal_handler+0x2a)
[0x7fadd7e1de5a] -> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0)
[0x7fadd7df562a] ->
/usr/lib/dovecot/modules/lib21_fts_solr_plugin.so(solr_connection_post_more+0x249)
[0x7fadd5f84929] ->
/usr/lib/dovecot/modules/lib21_fts_solr_plugin.so(+0x4af7)
[0x7fadd5f81af7] ->
/usr/lib/dovecot/modules/lib20_fts_plugin.so(fts_build_mail+0x3bd)
[0x7fadd65a6e5d] ->
/usr/lib/dovecot/modules/lib20_fts_plugin.so(+0xc570) [0x7fadd65ab570]
-> doveadm() [0x412ce1] -> doveadm() [0x40fbd2] -> doveadm() [0x4105db]
-> doveadm(doveadm_mail_try_run+0x161) [0x410a41] -> doveadm(main+0x375)
[0x40f7e5] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xfd)
[0x7fadd7a71ead] -> doveadm() [0x40f9d1]
Abandon


and catalina.out:

GRAVE: org.apache.solr.common.SolrException: Illegal character
((CTRL-CHAR, code 4))
 at [row,col {unknown-source}]: [533,500]
        at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:81)
        at
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:58)
        at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:1376)
        at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:365)
        at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:260)
        at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
        at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
        at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
        at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
        at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:602)
        at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
        at java.lang.Thread.run(Unknown Source)
Caused by: com.ctc.wstx.exc.WstxUnexpectedCharException: Illegal
character ((CTRL-CHAR, code 4))
 at [row,col {unknown-source}]: [533,500]
        at
com.ctc.wstx.sr.StreamScanner.throwInvalidSpace(StreamScanner.java:675)
        at
com.ctc.wstx.sr.BasicStreamReader.readTextSecondary(BasicStreamReader.java:4668)
        at
com.ctc.wstx.sr.BasicStreamReader.readCoalescedText(BasicStreamReader.java:4126)
        at
com.ctc.wstx.sr.BasicStreamReader.finishToken(BasicStreamReader.java:3701)
        at
com.ctc.wstx.sr.BasicStreamReader.safeFinishToken(BasicStreamReader.java:3649)
        at
com.ctc.wstx.sr.BasicStreamReader.getText(BasicStreamReader.java:809)
        at org.apache.solr.handler.XMLLoader.readDoc(XMLLoader.java:315)
        at
org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:156)
        at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:79)
        ... 17 more

31 juil. 2012 21:30:25 org.apache.solr.core.SolrCore execute
INFO: [] webapp=/solr path=/update params={} status=400 QTime=1



>> doveadm(clxx at spamguard.fr): Panic: file solr-connection.c: line 545
>> (solr_connection_post_more): assertion failed: (maxfd >= 0)
> I guess related to the previous error, but I'm not really sure why it happens either.
>

thanks again for your help

Best Regards,
Tonio Buonaguidi

More information about the dovecot mailing list