[Dovecot] fts solr : out of memory

[Matthew Powell]

On 2012-07-28, at 11.21, tonio at starbridge.org wrote:

> Fatal: master: service(indexer-worker): child 9582 returned error 83
> (Out of memory (service indexer-worker { vsz_limit=256 MB }, you may
> need to increase it))

This looks very much like an issue I've been having with fts-lucene. Indexing works (or at least completes without errors) in 2.1.7, but runs out of memory in 2.1.8.

doveadm -v index -u username '*' crashes with:

doveadm(username): Fatal: pool_system_realloc(536870912): Out of memory
doveadm(username): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x41d31) [0xb7699d31] -> /usr/lib/dovecot/libdovecot.so.0(default_fatal_handler+0x47) [0xb7699e47] -> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0xb766c958] -> /usr/lib/dovecot/libdovecot.so.0(+0x5420e) [0xb76ac20e] -> /usr/lib/dovecot/libdovecot.so.0(+0x3e358) [0xb7696358] -> /usr/lib/dovecot/libdovecot.so.0(buffer_get_space_unsafe+0x72) [0xb76967f2] -> /usr/lib/dovecot/libdovecot.so.0(buffer_append_space_unsafe+0x2b) [0xb769691b] -> /usr/lib/dovecot/libdovecot.so.0(charset_to_utf8+0xe8) [0xb768c3f8] -> /usr/lib/dovecot/libdovecot.so.0(message_decoder_decode_next_block+0x3dd) [0xb767fa4d] -> /usr/lib/dovecot/modules/lib20_fts_plugin.so(fts_build_mail+0x11c) [0xb732881c] -> /usr/lib/dovecot/modules/lib20_fts_plugin.so(+0xb256) [0xb732e256] -> /usr/lib/dovecot/libdovecot-storage.so.0(mail_precache+0xf) [0xb774754f] -> doveadm() [0x805736e] -> doveadm() [0x805422d] -> doveadm() [0x8054aad] -> doveadm(doveadm_mail_try_run+0x12e) [0x8054fae] -> doveadm(main+0x3e4) [0x805bf24] -> /lib/i686/cmov/libc.so.6(__libc_start_main+0xe6) [0xb7527ca6] -> doveadm() [0x8053c11]

The problem occurs repeatably with specific messages. It looks as though the crash happens in lib-charset/charset-iconv.c and the problem was introduced by this change: http://hg.dovecot.org/dovecot-2.1/rev/0fde692cb565. If I remove that one change from 2.1.8, I can index the affected messages again.

I've posted an example message that triggers the crash here: https://www.atom.net/dovecot/testcase1.eml. That's a very contrived test case, but I've encountered the same behaviour with a real message.

-- 

Matthew Powell                                          matthew at atom.net