[Dovecot] Dovecot Solr Panic
Hello Everyone,
I have a small base of users (30), but a lot of emails.
I have an error again when I am indexing a virtual folder with a large number of folders. I appreciate this is a special case, but I am using dovecot and solr as it is, according to the documentation, the favourite way.
One user is using a large number of archives sub folders, by years, months and subfolders, since 5 years. I am convincing this user to give up this organisation, but in the mean time, I hope to solve this problem.
The virtual folder, called Archives, references all the sub folders in the Archive folder. There is about 380 folders, and worst of all, 55k characters with their names concatenated.
On solr, I have increased the value of headerBufferSize to a huge value, but I have still this error.
Anyway, this is the error I have when I try to index this virtual folder:
---8<--------------------------------------------------------------------------------------------------------------------------------------- root@lapetus:/mnt/emails/indiefield.co.uk/jsmith/Mails/Virtual# doveadm -v index -u 'john smith' 'Search/Archives' doveadm(john smith): Info: Search/Archives: Caching mails seq=109580..127362 1800/17783doveadm(john smith): Panic: file solr-connection.c: line 416 (solr_connection_select): assertion failed: (!conn->posting) doveadm(john smith): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x4414a) [0x7fcb7b5b014a] -> /usr/lib/dovecot/libdovecot.so.0(default_fatal_handler+0x2a) [0x7fcb7b5b020a] -> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7fcb7b58782a] -> /usr/lib/dovecot/modules/lib21_fts_solr_plugin.so(+0x754a) [0x7fcb790fa54a] -> /usr/lib/dovecot/modules/lib21_fts_solr_plugin.so(+0x4037) [0x7fcb790f7037] -> /usr/lib/dovecot/modules/lib20_fts_plugin.so(fts_backend_get_last_uid+0x5a) [0x7fcb79b2f66a] -> /usr/lib/dovecot/modules/lib20_fts_plugin.so(+0xc1fd) [0x7fcb79b341fd] -> /usr/lib/dovecot/modules/lib20_fts_plugin.so(+0xc689) [0x7fcb79b34689] -> /usr/lib/dovecot/modules/lib20_fts_plugin.so(+0xc51e) [0x7fcb79b3451e] -> doveadm(+0x16719) [0x7fcb7bf71719] -> doveadm(+0x13346) [0x7fcb7bf6e346] -> doveadm(+0x13e21) [0x7fcb7bf6ee21] -> doveadm(doveadm_mail_try_run+0x161) [0x7fcb7bf6f2e1] -> doveadm(main+0x3d1) [0x7fcb7bf6def1] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xfd) [0x7fcb7b200ead] -> doveadm(+0x13125) [0x7fcb7bf6e125] Aborted ---8<---------------------------------------------------------------------------------------------------------------------------------------
On the solr side:
---8<--------------------------------------------------------------------------------------------------------------------------------------- 28-Apr-2013 17:13:33 org.apache.solr.update.processor.LogUpdateProcessor finish INFO: {add=[1/70af772eafc17c51fb350000812a8a77/john smith@indiefield.co.uk, 2/70af772eafc17c51fb350000812a8a77/john smith@indiefield.co.uk, 3/70af772eafc17c51fb350000812a8a 77/john smith@indiefield.co.uk, 4/70af772eafc17c51fb350000812a8a77/john smith@indiefield.co.uk, 5/70af772eafc17c51fb350000812a8a77/john smith@indiefield.co.uk, 6/70af772ea fc17c51fb350000812a8a77/john smith@indiefield.co.uk, 7/70af772eafc17c51fb350000812a8a77/john smith@indiefield.co.uk, 8/70af772eafc17c51fb350000812a8a77/john smith@indiefie ld.co.uk, ... (8 added)]} 0 19141 28-Apr-2013 17:13:33 org.apache.solr.common.SolrException log SEVERE: org.apache.solr.common.SolrException: ParseError at [row,col]:[1188418,131] Message: Premature end of file. at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:72) at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:54) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1317) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:943) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:843) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Caused by: javax.xml.stream.XMLStreamException: ParseError at [row,col]:[1188418,131] Message: Premature end of file. at com.sun.org.apache.xerces.internal.impl.XMLStreamReaderImpl.next(XMLStreamReaderImpl.java:592) at org.apache.solr.handler.XMLLoader.readDoc(XMLLoader.java:273) at org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:138) at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:69) ... 22 more ---8<---------------------------------------------------------------------------------------------------------------------------------------
Kind regards, Andre Rodier
Sorry,
I checked my dovecot version, and it has not been updated by Debian yet.
I will check if I can find a method to update without breaking dependancies.
André
On 28/04/13 17:42, Andre Rodier wrote:
Hello Everyone,
I have a small base of users (30), but a lot of emails.
I have an error again when I am indexing a virtual folder with a large number of folders. I appreciate this is a special case, but I am using dovecot and solr as it is, according to the documentation, the favourite way.
One user is using a large number of archives sub folders, by years, months and subfolders, since 5 years. I am convincing this user to give up this organisation, but in the mean time, I hope to solve this problem.
The virtual folder, called Archives, references all the sub folders in the Archive folder. There is about 380 folders, and worst of all, 55k characters with their names concatenated.
On solr, I have increased the value of headerBufferSize to a huge value, but I have still this error.
Anyway, this is the error I have when I try to index this virtual folder:
---8<--------------------------------------------------------------------------------------------------------------------------------------- root@lapetus:/mnt/emails/indiefield.co.uk/jsmith/Mails/Virtual# doveadm -v index -u 'john smith' 'Search/Archives' doveadm(john smith): Info: Search/Archives: Caching mails seq=109580..127362 1800/17783doveadm(john smith): Panic: file solr-connection.c: line 416 (solr_connection_select): assertion failed: (!conn->posting) doveadm(john smith): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x4414a) [0x7fcb7b5b014a] -> /usr/lib/dovecot/libdovecot.so.0(default_fatal_handler+0x2a) [0x7fcb7b5b020a] -> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7fcb7b58782a] -> /usr/lib/dovecot/modules/lib21_fts_solr_plugin.so(+0x754a) [0x7fcb790fa54a] -> /usr/lib/dovecot/modules/lib21_fts_solr_plugin.so(+0x4037) [0x7fcb790f7037] -> /usr/lib/dovecot/modules/lib20_fts_plugin.so(fts_backend_get_last_uid+0x5a) [0x7fcb79b2f66a] -> /usr/lib/dovecot/modules/lib20_fts_plugin.so(+0xc1fd) [0x7fcb79b341fd] -> /usr/lib/dovecot/modules/lib20_fts_plugin.so(+0xc689) [0x7fcb79b34689] -> /usr/lib/dovecot/modules/lib20_fts_plugin.so(+0xc51e) [0x7fcb79b3451e] -> doveadm(+0x16719) [0x7fcb7bf71719] -> doveadm(+0x13346) [0x7fcb7bf6e346] -> doveadm(+0x13e21) [0x7fcb7bf6ee21] -> doveadm(doveadm_mail_try_run+0x161) [0x7fcb7bf6f2e1] -> doveadm(main+0x3d1) [0x7fcb7bf6def1] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xfd) [0x7fcb7b200ead] -> doveadm(+0x13125) [0x7fcb7bf6e125] Aborted ---8<---------------------------------------------------------------------------------------------------------------------------------------
On the solr side:
---8<--------------------------------------------------------------------------------------------------------------------------------------- 28-Apr-2013 17:13:33 org.apache.solr.update.processor.LogUpdateProcessor finish INFO: {add=[1/70af772eafc17c51fb350000812a8a77/john smith@indiefield.co.uk, 2/70af772eafc17c51fb350000812a8a77/john smith@indiefield.co.uk, 3/70af772eafc17c51fb350000812a8a 77/john smith@indiefield.co.uk, 4/70af772eafc17c51fb350000812a8a77/john smith@indiefield.co.uk, 5/70af772eafc17c51fb350000812a8a77/john smith@indiefield.co.uk, 6/70af772ea fc17c51fb350000812a8a77/john smith@indiefield.co.uk, 7/70af772eafc17c51fb350000812a8a77/john smith@indiefield.co.uk, 8/70af772eafc17c51fb350000812a8a77/john smith@indiefie ld.co.uk, ... (8 added)]} 0 19141 28-Apr-2013 17:13:33 org.apache.solr.common.SolrException log SEVERE: org.apache.solr.common.SolrException: ParseError at [row,col]:[1188418,131] Message: Premature end of file. at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:72) at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:54) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1317) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:943) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:843) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Caused by: javax.xml.stream.XMLStreamException: ParseError at [row,col]:[1188418,131] Message: Premature end of file. at com.sun.org.apache.xerces.internal.impl.XMLStreamReaderImpl.next(XMLStreamReaderImpl.java:592) at org.apache.solr.handler.XMLLoader.readDoc(XMLLoader.java:273) at org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:138) at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:69) ... 22 more ---8<---------------------------------------------------------------------------------------------------------------------------------------
Kind regards, Andre Rodier
Am 29.04.2013 11:08, schrieb Andre Rodier:
Sorry,
I checked my dovecot version, and it has not been updated by Debian yet.
I will check if I can find a method to update without breaking dependancies.
André
perhaps look at
http://wiki.dovecot.org/PrebuiltBinaries#Automatically_Built_Packages
leads to rep
http://xi.rename-it.nl/debian/
Best Regards MfG Robert Schetterer
-- [*] sys4 AG
http://sys4.de, +49 (89) 30 90 46 64 Franziskanerstraße 15, 81669 München
Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263 Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer Aufsichtsratsvorsitzender: Florian Kirstein
Thanks Robert,
I was not aware of this distribution.
ATB, André
On 29/04/13 10:21, Robert Schetterer wrote:
Am 29.04.2013 11:08, schrieb Andre Rodier:
Sorry,
I checked my dovecot version, and it has not been updated by Debian yet.
I will check if I can find a method to update without breaking dependancies.
André
perhaps look at
http://wiki.dovecot.org/PrebuiltBinaries#Automatically_Built_Packages
leads to rep
http://xi.rename-it.nl/debian/
Best Regards MfG Robert Schetterer
participants (3)
-
Andre Rodier
-
André Rodier
-
Robert Schetterer