[Dovecot] fts solr : out of memory

[tonio at starbridge.org]

Le 28/07/2012 18:09, Matthew Powell a écrit :
> On 2012-07-28, at 11.21, tonio at starbridge.org wrote:
>
>> Fatal: master: service(indexer-worker): child 9582 returned error 83
>> (Out of memory (service indexer-worker { vsz_limit=256 MB }, you may
>> need to increase it))
> This looks very much like an issue I've been having with fts-lucene. Indexing works (or at least completes without errors) in 2.1.7, but runs out of memory in 2.1.8.
>
> doveadm -v index -u username '*' crashes with:
>
> doveadm(username): Fatal: pool_system_realloc(536870912): Out of memory
> doveadm(username): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x41d31) [0xb7699d31] -> /usr/lib/dovecot/libdovecot.so.0(default_fatal_handler+0x47) [0xb7699e47] -> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0xb766c958] -> /usr/lib/dovecot/libdovecot.so.0(+0x5420e) [0xb76ac20e] -> /usr/lib/dovecot/libdovecot.so.0(+0x3e358) [0xb7696358] -> /usr/lib/dovecot/libdovecot.so.0(buffer_get_space_unsafe+0x72) [0xb76967f2] -> /usr/lib/dovecot/libdovecot.so.0(buffer_append_space_unsafe+0x2b) [0xb769691b] -> /usr/lib/dovecot/libdovecot.so.0(charset_to_utf8+0xe8) [0xb768c3f8] -> /usr/lib/dovecot/libdovecot.so.0(message_decoder_decode_next_block+0x3dd) [0xb767fa4d] -> /usr/lib/dovecot/modules/lib20_fts_plugin.so(fts_build_mail+0x11c) [0xb732881c] -> /usr/lib/dovecot/modules/lib20_fts_plugin.so(+0xb256) [0xb732e256] -> /usr/lib/dovecot/libdovecot-storage.so.0(mail_precache+0xf) [0xb774754f] -> doveadm() [0x805736e] -> doveadm() [0x805422d] -> doveadm() [0x8054aad] -> doveadm(doveadm_mail_try_run+0x12e) [0x8054fae] -> doveadm(main+0x3e4) [0x805bf24] -> /lib/i686/cmov/libc.so.6(__libc_start_main+0xe6) [0xb7527ca6] -> doveadm() [0x8053c11]
>
> The problem occurs repeatably with specific messages. It looks as though the crash happens in lib-charset/charset-iconv.c and the problem was introduced by this change: http://hg.dovecot.org/dovecot-2.1/rev/0fde692cb565. If I remove that one change from 2.1.8, I can index the affected messages again.
>
> I've posted an example message that triggers the crash here: https://www.atom.net/dovecot/testcase1.eml. That's a very contrived test case, but I've encountered the same behaviour with a real message.
>
Hi Matthew,
thanks for your answer.
most of our email contains French accents, so may be it's related.

I've forgot to post the complete log message for this error earlier:

Jul 27 19:01:00 mx2 dovecot: indexer-worker(cxx at crxx.fr): Fatal:
pool_system_realloc(268435456): Out of memory
Jul 27 19:01:00 mx2 dovecot: indexer-worker(cxx at crxx.fr): Error: Raw
backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x43e4a) [0x7fa49c7e4e4a]
-> /usr/lib/dovecot/libdovecot.so.0(+0x43e8e) [0x7fa49c7e4e8e] ->
/usr/lib/dovecot/libdovecot.so.0(i_error+0) [0x7fa49c7bc784] ->
/usr/lib/dovecot/libdovecot.so.0(+0x542f0) [0x7fa49c7f52f0] ->
/usr/lib/dovecot/libdovecot.so.0(+0x40fe5) [0x7fa49c7e1fe5] ->
/usr/lib/dovecot/libdovecot.so.0(buffer_get_space_unsafe+0x6b)
[0x7fa49c7e297b] ->
/usr/lib/dovecot/libdovecot.so.0(charset_to_utf8+0xe7) [0x7fa49c7d9867]
->
/usr/lib/dovecot/libdovecot.so.0(message_decoder_decode_next_block+0x3b8) [0x7fa49c7ce9d8]
-> /usr/lib/dovecot/modules/lib20_fts_plugin.so(fts_build_mail+0x194)
[0x7fa49af6dc34] ->
/usr/lib/dovecot/modules/lib20_fts_plugin.so(+0xc570) [0x7fa49af72570]
-> dovecot/indexer-worker() [0x402544] ->
/usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x36) [0x7fa49c7f1b96]
-> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0xa7)
[0x7fa49c7f2827] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x28)
[0x7fa49c7f15a8] ->
/usr/lib/dovecot/libdovecot.so.0(master_service_run+0x13)
[0x7fa49c7de2c3] -> dovecot/indexer-worker(main+0xe2) [0x402002] ->
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xfd) [0x7fa49c438ead]
-> dovecot/indexer-worker() [0x4020a1]
Jul 27 19:01:00 mx2 dovecot: indexer: Error: Indexer worker
disconnected, discarding 1 requests for cxx at crxx.fr
Jul 27 19:01:00 mx2 dovecot: imap(cxx at crxx.fr): Error: indexer failed to
index mailbox INBOX
Jul 27 19:01:00 mx2 dovecot: indexer-worker(cxx at crxx): Fatal: master:
service(indexer-worker): child 9582 returned error 83 (Out of memory
(service indexer-worker { vsz_limit=256 MB }, you may need to increase it))


Hope this helps !