[Dovecot] Possible mdbox corruption

[Daniel L. Miller]

On 1/6/2012 2:57 AM, Timo Sirainen wrote:
> On 6.1.2012, at 12.55, Timo Sirainen wrote:
>
>>> Jan  4 05:17:17 bubba dovecot: master: Error: service(indexer-worker): child 10896 returned error 83 (Out of memory (vsz_limit=256 MB, you may need to increase it))
>>> Jan  4 06:17:17 bubba dovecot: indexer-worker(user1 at domain.com): Fatal: pool_system_realloc(134217728): Out of memory
>> The problem is clearly that index-worker's vsz_limit is too low. Increase it (or default_vsz_limit).
> Although the source of the out-of-memory
>
> /usr/local/lib/dovecot/libdovecot.so.0(buffer_write+0x7c) [0x7f0ec1a550ec] ->  /usr/local/lib/dovecot/lib21_fts_solr_plugin.so(+0x3292) [0x7f0ec024f292] ->
>
> is something that shouldn't really be happening. I guess the Solr plugin wastes memory unnecessarily, I'll see what I can do about it. But for now just increase vsz limit.
>
I set default_vsz_limit = 1024M.  Those errors appear gone - but I do 
have messages like:

Jan  6 09:22:42 bubba dovecot: indexer-worker(user1 at domain.com): Error: 
fts_solr: Indexing failed: 400 Illegal character ((CTRL-CHAR, code 18))  
at [row,col {unknown-source}]: [482765,16]
Jan  6 09:22:42 bubba dovecot: indexer-worker: Error:

Google seems to indicate that Solr cannot handle "invalid" characters - 
and that it is the responsibility of the calling program to strip out 
such.  A quick search shows me a both an individual character comparison 
in Java and a regex used for the purpose.  Is there any "illegal 
character protection" in the Dovecot Solr plugin?

-- 
Daniel