[Dovecot] Dovecot Solr Panic

[Andre Rodier]

Hello Everyone,

I have a small base of users (30), but a lot of emails.

I have an error again when I am indexing a virtual folder with a large number of folders.
I appreciate this is a special case, but I am using dovecot and solr as it is, according to the documentation, the favourite way.

One user is using a large number of archives sub folders, by years, months and subfolders, since 5 years. I am convincing this user to give up this
organisation, but in the mean time, I hope to solve this problem.

The virtual folder, called Archives, references all the sub folders in the Archive folder. There is about 380 folders, and worst of all, 55k characters with
their names concatenated.

On solr, I have increased the value of headerBufferSize to a huge value, but I have still this error.

Anyway, this is the error I have when I try to index this virtual folder:

---8<---------------------------------------------------------------------------------------------------------------------------------------
root at lapetus:/mnt/emails/indiefield.co.uk/jsmith/Mails/Virtual# doveadm -v index -u 'john smith' 'Search/Archives'
doveadm(john smith): Info: Search/Archives: Caching mails seq=109580..127362
1800/17783doveadm(john smith): Panic: file solr-connection.c: line 416 (solr_connection_select): assertion failed: (!conn->posting)
doveadm(john smith): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x4414a) [0x7fcb7b5b014a] ->
/usr/lib/dovecot/libdovecot.so.0(default_fatal_handler+0x2a) [0x7fcb7b5b020a] -> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7fcb7b58782a] ->
/usr/lib/dovecot/modules/lib21_fts_solr_plugin.so(+0x754a) [0x7fcb790fa54a] -> /usr/lib/dovecot/modules/lib21_fts_solr_plugin.so(+0x4037) [0x7fcb790f7037] ->
/usr/lib/dovecot/modules/lib20_fts_plugin.so(fts_backend_get_last_uid+0x5a) [0x7fcb79b2f66a] -> /usr/lib/dovecot/modules/lib20_fts_plugin.so(+0xc1fd)
[0x7fcb79b341fd] -> /usr/lib/dovecot/modules/lib20_fts_plugin.so(+0xc689) [0x7fcb79b34689] -> /usr/lib/dovecot/modules/lib20_fts_plugin.so(+0xc51e)
[0x7fcb79b3451e] -> doveadm(+0x16719) [0x7fcb7bf71719] -> doveadm(+0x13346) [0x7fcb7bf6e346] -> doveadm(+0x13e21) [0x7fcb7bf6ee21] ->
doveadm(doveadm_mail_try_run+0x161) [0x7fcb7bf6f2e1] -> doveadm(main+0x3d1) [0x7fcb7bf6def1] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xfd)
[0x7fcb7b200ead] -> doveadm(+0x13125) [0x7fcb7bf6e125]
Aborted
---8<---------------------------------------------------------------------------------------------------------------------------------------

On the solr side:

---8<---------------------------------------------------------------------------------------------------------------------------------------
28-Apr-2013 17:13:33 org.apache.solr.update.processor.LogUpdateProcessor finish
INFO: {add=[1/70af772eafc17c51fb350000812a8a77/john smith at indiefield.co.uk, 2/70af772eafc17c51fb350000812a8a77/john smith at indiefield.co.uk, 3/70af772eafc17c51fb350000812a8a
77/john smith at indiefield.co.uk, 4/70af772eafc17c51fb350000812a8a77/john smith at indiefield.co.uk, 5/70af772eafc17c51fb350000812a8a77/john smith at indiefield.co.uk, 6/70af772ea
fc17c51fb350000812a8a77/john smith at indiefield.co.uk, 7/70af772eafc17c51fb350000812a8a77/john smith at indiefield.co.uk, 8/70af772eafc17c51fb350000812a8a77/john smith at indiefie
ld.co.uk, ... (8 added)]} 0 19141
28-Apr-2013 17:13:33 org.apache.solr.common.SolrException log
SEVERE: org.apache.solr.common.SolrException: ParseError at [row,col]:[1188418,131]
Message: Premature end of file.
        at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:72)
        at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:54)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:1317)
        at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
        at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)
        at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
        at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
        at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
        at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
        at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
        at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
        at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
        at org.mortbay.jetty.Server.handle(Server.java:326)
        at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
        at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:943)
        at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:843)
        at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
        at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
        at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
        at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Caused by: javax.xml.stream.XMLStreamException: ParseError at [row,col]:[1188418,131]
Message: Premature end of file.
        at com.sun.org.apache.xerces.internal.impl.XMLStreamReaderImpl.next(XMLStreamReaderImpl.java:592)
        at org.apache.solr.handler.XMLLoader.readDoc(XMLLoader.java:273)
        at org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:138)
        at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:69)
        ... 22 more
---8<---------------------------------------------------------------------------------------------------------------------------------------

Kind regards,
Andre Rodier