[Dovecot] Possible mdbox corruption
I thought I had cleared out the corruption I had before - perhaps I was
mistaken. What steps should I take to help locate these issues?
Currently using 2.1rc1. I see the following errors in my logs,
including out of memory and message size issues (at 15:30):
Jan 4 05:17:17 bubba dovecot: indexer-worker(user1@domain.com): Error: Raw backtrace: /usr/local/lib/dovecot/libdovecot.so.0(+0x3ed0a) [0x7f6e17cbfd0a] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x3ed56) [0x7f6e17cbfd56] -> /usr/local/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7f6e17c98d08] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x4f310) [0x7f6e17cd0310] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x3b965) [0x7f6e17cbc965] -> /usr/local/lib/dovecot/libdovecot.so.0(buffer_write+0x7c) [0x7f6e17cbd0ec] -> /usr/local/lib/dovecot/lib21_fts_solr_plugin.so(+0x3292) [0x7f6e164b7292] -> /usr/local/lib/dovecot/lib21_fts_solr_plugin.so(+0x3a97) [0x7f6e164b7a97] -> /usr/local/lib/dovecot/lib20_fts_plugin.so(fts_backend_update_set_build_key+0x2c) [0x7f6e166c4abc] -> /usr/local/lib/dovecot/lib20_fts_plugin.so(fts_build_mail+0x2d1) [0x7f6e166c5561] -> /usr/local/lib/dovecot/lib20_fts_plugin.so(+0xc630) [0x7f6e166ca630] -> dovecot/indexer-worker user1@domain.com Sent - 5500/7510 [0x40245f] -> dovecot/indexer-worker [user1@domain.com Sent pool_system_realloc(134217728): Out of memory
- 5500/7510]() [0x4027dd] -> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x36) [0x7f6e17ccc0f6] -> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x9f) [0x7f6e17ccd17f] -> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_run+0x28) [0x7f6e17ccc098] -> /usr/local/lib/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7f6e17cb9123] -> dovecot/indexer-worker user1@domain.com Sent - 5500/7510 [0x401f29] -> /lib/libc.so.6(__libc_start_main+0xfe) [0x7f6e1791cd8e] -> dovecot/indexer-worker user1@domain.com Sent - 5500/7510 [0x401d19] Jan 4 05:17:17 bubba dovecot: indexer: Error: Indexer worker disconnected, discarding 1 requests for user1@domain.com Jan 4 05:17:17 bubba dovecot: master: Error: service(indexer-worker): child 10896 returned error 83 (Out of memory (vsz_limit=256 MB, you may need to increase it)) Jan 4 06:17:17 bubba dovecot: indexer-worker(user1@domain.com): Fatal:
Jan 4 06:17:17 bubba dovecot: indexer-worker(user1@domain.com): Error: Raw backtrace: /usr/local/lib/dovecot/libdovecot.so.0(+0x3ed0a) [0x7f0ec1a57d0a] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x3ed56) [0x7f0ec1a57d56] -> /usr/local/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7f0ec1a30d08] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x4f310) [0x7f0ec1a68310] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x3b965) [0x7f0ec1a54965] -> /usr/local/lib/dovecot/libdovecot.so.0(buffer_write+0x7c) [0x7f0ec1a550ec] -> /usr/local/lib/dovecot/lib21_fts_solr_plugin.so(+0x3292) [0x7f0ec024f292] -> /usr/local/lib/dovecot/lib21_fts_solr_plugin.so(+0x3a97) [0x7f0ec024fa97] -> /usr/local/lib/dovecot/lib20_fts_plugin.so(fts_backend_update_set_build_key+0x2c) [0x7f0ec045cabc] -> /usr/local/lib/dovecot/lib20_fts_plugin.so(fts_build_mail+0x2d1) [0x7f0ec045d561] -> /usr/local/lib/dovecot/lib20_fts_plugin.so(+0xc630) [0x7f0ec0462630] -> dovecot/indexer-worker user1@domain.com Sent - 5500/7510 [0x40245f] -> dovecot/indexer-worker [user1@domain.com Sent pool_system_realloc(134217728): Out of memory
- 5500/7510]() [0x4027dd] -> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x36) [0x7f0ec1a640f6] -> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x9f) [0x7f0ec1a6517f] -> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_run+0x28) [0x7f0ec1a64098] -> /usr/local/lib/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7f0ec1a51123] -> dovecot/indexer-worker user1@domain.com Sent - 5500/7510 [0x401f29] -> /lib/libc.so.6(__libc_start_main+0xfe) [0x7f0ec16b4d8e] -> dovecot/indexer-worker user1@domain.com Sent - 5500/7510 [0x401d19] Jan 4 06:17:17 bubba dovecot: indexer: Error: Indexer worker disconnected, discarding 1 requests for user1@domain.com Jan 4 06:17:17 bubba dovecot: master: Error: service(indexer-worker): child 11941 returned error 83 (Out of memory (vsz_limit=256 MB, you may need to increase it)) Jan 4 07:17:18 bubba dovecot: indexer-worker(user1@domain.com): Fatal:
Jan 4 07:17:18 bubba dovecot: indexer-worker(user1@domain.com): Error: Raw backtrace: /usr/local/lib/dovecot/libdovecot.so.0(+0x3ed0a) [0x7faed4e56d0a] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x3ed56) [0x7faed4e56d56] -> /usr/local/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7faed4e2fd08] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x4f310) [0x7faed4e67310] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x3b965) [0x7faed4e53965] -> /usr/local/lib/dovecot/libdovecot.so.0(buffer_write+0x7c) [0x7faed4e540ec] -> /usr/local/lib/dovecot/lib21_fts_solr_plugin.so(+0x3292) [0x7faed364e292] -> /usr/local/lib/dovecot/lib21_fts_solr_plugin.so(+0x3a97) [0x7faed364ea97] -> /usr/local/lib/dovecot/lib20_fts_plugin.so(fts_backend_update_set_build_key+0x2c) [0x7faed385babc] -> /usr/local/lib/dovecot/lib20_fts_plugin.so(fts_build_mail+0x2d1) [0x7faed385c561] -> /usr/local/lib/dovecot/lib20_fts_plugin.so(+0xc630) [0x7faed3861630] -> dovecot/indexer-worker user1@domain.com Sent - 5500/7510 [0x40245f] -> dovecot/indexer-worker [user1@domain.com Sent pool_system_realloc(134217728): Out of memory
- 5500/7510]() [0x4027dd] -> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x36) [0x7faed4e630f6] -> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x9f) [0x7faed4e6417f] -> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_run+0x28) [0x7faed4e63098] -> /usr/local/lib/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7faed4e50123] -> dovecot/indexer-worker user1@domain.com Sent - 5500/7510 [0x401f29] -> /lib/libc.so.6(__libc_start_main+0xfe) [0x7faed4ab3d8e] -> dovecot/indexer-worker user1@domain.com Sent - 5500/7510 [0x401d19] Jan 4 07:17:18 bubba dovecot: indexer: Error: Indexer worker disconnected, discarding 1 requests for user1@domain.com Jan 4 07:17:18 bubba dovecot: master: Error: service(indexer-worker): child 13299 returned error 83 (Out of memory (vsz_limit=256 MB, you may need to increase it)) Jan 4 08:17:17 bubba dovecot: indexer-worker(user1@domain.com): Fatal:
Jan 4 08:17:17 bubba dovecot: indexer-worker(user1@domain.com): Error: Raw backtrace: /usr/local/lib/dovecot/libdovecot.so.0(+0x3ed0a) [0x7ffd84382d0a] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x3ed56) [0x7ffd84382d56] -> /usr/local/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7ffd8435bd08] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x4f310) [0x7ffd84393310] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x3b965) [0x7ffd8437f965] -> /usr/local/lib/dovecot/libdovecot.so.0(buffer_write+0x7c) [0x7ffd843800ec] -> /usr/local/lib/dovecot/lib21_fts_solr_plugin.so(+0x3292) [0x7ffd82b7a292] -> /usr/local/lib/dovecot/lib21_fts_solr_plugin.so(+0x3a97) [0x7ffd82b7aa97] -> /usr/local/lib/dovecot/lib20_fts_plugin.so(fts_backend_update_set_build_key+0x2c) [0x7ffd82d87abc] -> /usr/local/lib/dovecot/lib20_fts_plugin.so(fts_build_mail+0x2d1) [0x7ffd82d88561] -> /usr/local/lib/dovecot/lib20_fts_plugin.so(+0xc630) [0x7ffd82d8d630] -> dovecot/indexer-worker user1@domain.com Sent - 5500/7510 [0x40245f] -> dovecot/indexer-worker [user1@domain.com Sent pool_system_realloc(134217728): Out of memory
- 5500/7510]() [0x4027dd] -> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x36) [0x7ffd8438f0f6] -> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x9f) [0x7ffd8439017f] -> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_run+0x28) [0x7ffd8438f098] -> /usr/local/lib/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7ffd8437c123] -> dovecot/indexer-worker user1@domain.com Sent - 5500/7510 [0x401f29] -> /lib/libc.so.6(__libc_start_main+0xfe) [0x7ffd83fdfd8e] -> dovecot/indexer-worker user1@domain.com Sent - 5500/7510 [0x401d19] Jan 4 08:17:17 bubba dovecot: indexer: Error: Indexer worker disconnected, discarding 1 requests for user1@domain.com Jan 4 08:17:17 bubba dovecot: master: Error: service(indexer-worker): child 14413 returned error 83 (Out of memory (vsz_limit=256 MB, you may need to increase it)) Jan 4 09:17:19 bubba dovecot: indexer-worker(user1@domain.com): Fatal:
Jan 4 09:17:19 bubba dovecot: indexer-worker(user1@domain.com): Error: Raw backtrace: /usr/local/lib/dovecot/libdovecot.so.0(+0x3ed0a) [0x7fb701bf5d0a] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x3ed56) [0x7fb701bf5d56] -> /usr/local/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7fb701bced08] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x4f310) [0x7fb701c06310] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x3b965) [0x7fb701bf2965] -> /usr/local/lib/dovecot/libdovecot.so.0(buffer_write+0x7c) [0x7fb701bf30ec] -> /usr/local/lib/dovecot/lib21_fts_solr_plugin.so(+0x3292) [0x7fb7003ed292] -> /usr/local/lib/dovecot/lib21_fts_solr_plugin.so(+0x3a97) [0x7fb7003eda97] -> /usr/local/lib/dovecot/lib20_fts_plugin.so(fts_backend_update_set_build_key+0x2c) [0x7fb7005faabc] -> /usr/local/lib/dovecot/lib20_fts_plugin.so(fts_build_mail+0x2d1) [0x7fb7005fb561] -> /usr/local/lib/dovecot/lib20_fts_plugin.so(+0xc630) [0x7fb700600630] -> dovecot/indexer-worker user1@domain.com Sent - 5500/7510 [0x40245f] -> dovecot/indexer-worker [user1@domain.com Sent pool_system_realloc(134217728): Out of memory
- 5500/7510]() [0x4027dd] -> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x36) [0x7fb701c020f6] -> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x9f) [0x7fb701c0317f] -> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_run+0x28) [0x7fb701c02098] -> /usr/local/lib/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7fb701bef123] -> dovecot/indexer-worker user1@domain.com Sent - 5500/7510 [0x401f29] -> /lib/libc.so.6(__libc_start_main+0xfe) [0x7fb701852d8e] -> dovecot/indexer-worker user1@domain.com Sent - 5500/7510 [0x401d19] Jan 4 09:17:19 bubba dovecot: indexer: Error: Indexer worker disconnected, discarding 1 requests for user1@domain.com Jan 4 09:17:19 bubba dovecot: master: Error: service(indexer-worker): child 15486 returned error 83 (Out of memory (vsz_limit=256 MB, you may need to increase it)) Jan 4 10:17:17 bubba dovecot: indexer-worker(user1@domain.com): Fatal:
Jan 4 10:17:17 bubba dovecot: indexer-worker(user1@domain.com): Error: Raw backtrace: /usr/local/lib/dovecot/libdovecot.so.0(+0x3ed0a) [0x7f8dc590ed0a] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x3ed56) [0x7f8dc590ed56] -> /usr/local/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7f8dc58e7d08] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x4f310) [0x7f8dc591f310] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x3b965) [0x7f8dc590b965] -> /usr/local/lib/dovecot/libdovecot.so.0(buffer_write+0x7c) [0x7f8dc590c0ec] -> /usr/local/lib/dovecot/lib21_fts_solr_plugin.so(+0x3292) [0x7f8dc4106292] -> /usr/local/lib/dovecot/lib21_fts_solr_plugin.so(+0x3a97) [0x7f8dc4106a97] -> /usr/local/lib/dovecot/lib20_fts_plugin.so(fts_backend_update_set_build_key+0x2c) [0x7f8dc4313abc] -> /usr/local/lib/dovecot/lib20_fts_plugin.so(fts_build_mail+0x2d1) [0x7f8dc4314561] -> /usr/local/lib/dovecot/lib20_fts_plugin.so(+0xc630) [0x7f8dc4319630] -> dovecot/indexer-worker user1@domain.com Sent - 5500/7510 [0x40245f] -> dovecot/indexer-worker [user1@domain.com Sent pool_system_realloc(134217728): Out of memory
- 5500/7510]() [0x4027dd] -> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x36) [0x7f8dc591b0f6] -> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x9f) [0x7f8dc591c17f] -> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_run+0x28) [0x7f8dc591b098] -> /usr/local/lib/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7f8dc5908123] -> dovecot/indexer-worker user1@domain.com Sent - 5500/7510 [0x401f29] -> /lib/libc.so.6(__libc_start_main+0xfe) [0x7f8dc556bd8e] -> dovecot/indexer-worker user1@domain.com Sent - 5500/7510 [0x401d19] Jan 4 10:17:17 bubba dovecot: indexer: Error: Indexer worker disconnected, discarding 1 requests for user1@domain.com Jan 4 10:17:17 bubba dovecot: master: Error: service(indexer-worker): child 16472 returned error 83 (Out of memory (vsz_limit=256 MB, you may need to increase it)) Jan 4 11:17:17 bubba dovecot: indexer-worker(user1@domain.com): Fatal:
Jan 4 11:17:17 bubba dovecot: indexer-worker(user1@domain.com): Error: Raw backtrace: /usr/local/lib/dovecot/libdovecot.so.0(+0x3ed0a) [0x7ff619c1dd0a] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x3ed56) [0x7ff619c1dd56] -> /usr/local/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7ff619bf6d08] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x4f310) [0x7ff619c2e310] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x3b965) [0x7ff619c1a965] -> /usr/local/lib/dovecot/libdovecot.so.0(buffer_write+0x7c) [0x7ff619c1b0ec] -> /usr/local/lib/dovecot/lib21_fts_solr_plugin.so(+0x3292) [0x7ff618415292] -> /usr/local/lib/dovecot/lib21_fts_solr_plugin.so(+0x3a97) [0x7ff618415a97] -> /usr/local/lib/dovecot/lib20_fts_plugin.so(fts_backend_update_set_build_key+0x2c) [0x7ff618622abc] -> /usr/local/lib/dovecot/lib20_fts_plugin.so(fts_build_mail+0x2d1) [0x7ff618623561] -> /usr/local/lib/dovecot/lib20_fts_plugin.so(+0xc630) [0x7ff618628630] -> dovecot/indexer-worker user1@domain.com Sent - 5500/7514 [0x40245f] -> dovecot/indexer-worker [user1@domain.com Sent pool_system_realloc(134217728): Out of memory
- 5500/7514]() [0x4027dd] -> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x36) [0x7ff619c2a0f6] -> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x9f) [0x7ff619c2b17f] -> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_run+0x28) [0x7ff619c2a098] -> /usr/local/lib/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7ff619c17123] -> dovecot/indexer-worker user1@domain.com Sent - 5500/7514 [0x401f29] -> /lib/libc.so.6(__libc_start_main+0xfe) [0x7ff61987ad8e] -> dovecot/indexer-worker user1@domain.com Sent - 5500/7514 [0x401d19] Jan 4 11:17:17 bubba dovecot: indexer: Error: Indexer worker disconnected, discarding 1 requests for user1@domain.com Jan 4 11:17:17 bubba dovecot: master: Error: service(indexer-worker): child 17522 returned error 83 (Out of memory (vsz_limit=256 MB, you may need to increase it)) Jan 4 12:17:17 bubba dovecot: indexer-worker(user1@domain.com): Fatal:
Jan 4 12:17:17 bubba dovecot: indexer-worker(user1@domain.com): Error: Raw backtrace: /usr/local/lib/dovecot/libdovecot.so.0(+0x3ed0a) [0x7ffd988c1d0a] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x3ed56) [0x7ffd988c1d56] -> /usr/local/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7ffd9889ad08] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x4f310) [0x7ffd988d2310] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x3b965) [0x7ffd988be965] -> /usr/local/lib/dovecot/libdovecot.so.0(buffer_write+0x7c) [0x7ffd988bf0ec] -> /usr/local/lib/dovecot/lib21_fts_solr_plugin.so(+0x3292) [0x7ffd970b9292] -> /usr/local/lib/dovecot/lib21_fts_solr_plugin.so(+0x3a97) [0x7ffd970b9a97] -> /usr/local/lib/dovecot/lib20_fts_plugin.so(fts_backend_update_set_build_key+0x2c) [0x7ffd972c6abc] -> /usr/local/lib/dovecot/lib20_fts_plugin.so(fts_build_mail+0x2d1) [0x7ffd972c7561] -> /usr/local/lib/dovecot/lib20_fts_plugin.so(+0xc630) [0x7ffd972cc630] -> dovecot/indexer-worker user1@domain.com Sent - 5500/7515 [0x40245f] -> dovecot/indexer-worker [user1@domain.com Sent pool_system_realloc(134217728): Out of memory
- 5500/7515]() [0x4027dd] -> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x36) [0x7ffd988ce0f6] -> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x9f) [0x7ffd988cf17f] -> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_run+0x28) [0x7ffd988ce098] -> /usr/local/lib/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7ffd988bb123] -> dovecot/indexer-worker user1@domain.com Sent - 5500/7515 [0x401f29] -> /lib/libc.so.6(__libc_start_main+0xfe) [0x7ffd9851ed8e] -> dovecot/indexer-worker user1@domain.com Sent - 5500/7515 [0x401d19] Jan 4 12:17:17 bubba dovecot: indexer: Error: Indexer worker disconnected, discarding 1 requests for user1@domain.com Jan 4 12:17:17 bubba dovecot: master: Error: service(indexer-worker): child 18498 returned error 83 (Out of memory (vsz_limit=256 MB, you may need to increase it)) Jan 4 13:17:16 bubba dovecot: indexer-worker(user1@domain.com): Fatal:
Jan 4 13:17:16 bubba dovecot: indexer-worker(user1@domain.com): Error: Raw backtrace: /usr/local/lib/dovecot/libdovecot.so.0(+0x3ed0a) [0x7f51b0163d0a] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x3ed56) [0x7f51b0163d56] -> /usr/local/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7f51b013cd08] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x4f310) [0x7f51b0174310] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x3b965) [0x7f51b0160965] -> /usr/local/lib/dovecot/libdovecot.so.0(buffer_write+0x7c) [0x7f51b01610ec] -> /usr/local/lib/dovecot/lib21_fts_solr_plugin.so(+0x3292) [0x7f51ae95b292] -> /usr/local/lib/dovecot/lib21_fts_solr_plugin.so(+0x3a97) [0x7f51ae95ba97] -> /usr/local/lib/dovecot/lib20_fts_plugin.so(fts_backend_update_set_build_key+0x2c) [0x7f51aeb68abc] -> /usr/local/lib/dovecot/lib20_fts_plugin.so(fts_build_mail+0x2d1) [0x7f51aeb69561] -> /usr/local/lib/dovecot/lib20_fts_plugin.so(+0xc630) [0x7f51aeb6e630] -> dovecot/indexer-worker user1@domain.com Sent - 5500/7515 [0x40245f] -> dovecot/indexer-worker [user1@domain.com Sent pool_system_realloc(134217728): Out of memory
- 5500/7515]() [0x4027dd] -> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x36) [0x7f51b01700f6] -> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x9f) [0x7f51b017117f] -> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_run+0x28) [0x7f51b0170098] -> /usr/local/lib/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7f51b015d123] -> dovecot/indexer-worker user1@domain.com Sent - 5500/7515 [0x401f29] -> /lib/libc.so.6(__libc_start_main+0xfe) [0x7f51afdc0d8e] -> dovecot/indexer-worker user1@domain.com Sent - 5500/7515 [0x401d19] Jan 4 13:17:16 bubba dovecot: indexer: Error: Indexer worker disconnected, discarding 1 requests for user1@domain.com Jan 4 13:17:16 bubba dovecot: master: Error: service(indexer-worker): child 19550 returned error 83 (Out of memory (vsz_limit=256 MB, you may need to increase it)) Jan 4 14:17:17 bubba dovecot: indexer-worker(user1@domain.com): Fatal:
Jan 4 14:17:17 bubba dovecot: indexer-worker(user1@domain.com): Error: Raw backtrace: /usr/local/lib/dovecot/libdovecot.so.0(+0x3ed0a) [0x7f423b546d0a] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x3ed56) [0x7f423b546d56] -> /usr/local/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7f423b51fd08] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x4f310) [0x7f423b557310] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x3b965) [0x7f423b543965] -> /usr/local/lib/dovecot/libdovecot.so.0(buffer_write+0x7c) [0x7f423b5440ec] -> /usr/local/lib/dovecot/lib21_fts_solr_plugin.so(+0x3292) [0x7f4239d3e292] -> /usr/local/lib/dovecot/lib21_fts_solr_plugin.so(+0x3a97) [0x7f4239d3ea97] -> /usr/local/lib/dovecot/lib20_fts_plugin.so(fts_backend_update_set_build_key+0x2c) [0x7f4239f4babc] -> /usr/local/lib/dovecot/lib20_fts_plugin.so(fts_build_mail+0x2d1) [0x7f4239f4c561] -> /usr/local/lib/dovecot/lib20_fts_plugin.so(+0xc630) [0x7f4239f51630] -> dovecot/indexer-worker user1@domain.com Sent - 5500/7516 [0x40245f] -> dovecot/indexer-worker [user1@domain.com Sent pool_system_realloc(134217728): Out of memory
- 5500/7516]() [0x4027dd] -> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x36) [0x7f423b5530f6] -> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x9f) [0x7f423b55417f] -> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_run+0x28) [0x7f423b553098] -> /usr/local/lib/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7f423b540123] -> dovecot/indexer-worker user1@domain.com Sent - 5500/7516 [0x401f29] -> /lib/libc.so.6(__libc_start_main+0xfe) [0x7f423b1a3d8e] -> dovecot/indexer-worker user1@domain.com Sent - 5500/7516 [0x401d19] Jan 4 14:17:17 bubba dovecot: master: Error: service(indexer-worker): child 20638 returned error 83 (Out of memory (vsz_limit=256 MB, you may need to increase it)) Jan 4 14:17:17 bubba dovecot: indexer: Error: Indexer worker disconnected, discarding 1 requests for user1@domain.com Jan 4 15:17:17 bubba dovecot: indexer-worker(user1@domain.com): Fatal:
Jan 4 15:17:17 bubba dovecot: indexer-worker(user1@domain.com): Error: Raw backtrace: /usr/local/lib/dovecot/libdovecot.so.0(+0x3ed0a) [0x7f3ab5e51d0a] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x3ed56) [0x7f3ab5e51d56] -> /usr/local/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7f3ab5e2ad08] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x4f310) [0x7f3ab5e62310] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x3b965) [0x7f3ab5e4e965] -> /usr/local/lib/dovecot/libdovecot.so.0(buffer_write+0x7c) [0x7f3ab5e4f0ec] -> /usr/local/lib/dovecot/lib21_fts_solr_plugin.so(+0x3292) [0x7f3ab4649292] -> /usr/local/lib/dovecot/lib21_fts_solr_plugin.so(+0x3a97) [0x7f3ab4649a97] -> /usr/local/lib/dovecot/lib20_fts_plugin.so(fts_backend_update_set_build_key+0x2c) [0x7f3ab4856abc] -> /usr/local/lib/dovecot/lib20_fts_plugin.so(fts_build_mail+0x2d1) [0x7f3ab4857561] -> /usr/local/lib/dovecot/lib20_fts_plugin.so(+0xc630) [0x7f3ab485c630] -> dovecot/indexer-worker user1@domain.com Sent - 5500/7517 [0x40245f] -> dovecot/indexer-worker [user1@domain.com Sent pool_system_realloc(134217728): Out of memory
- 5500/7517]() [0x4027dd] -> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x36) [0x7f3ab5e5e0f6] -> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x9f) [0x7f3ab5e5f17f] -> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_run+0x28) [0x7f3ab5e5e098] -> /usr/local/lib/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7f3ab5e4b123] -> dovecot/indexer-worker user1@domain.com Sent - 5500/7517 [0x401f29] -> /lib/libc.so.6(__libc_start_main+0xfe) [0x7f3ab5aaed8e] -> dovecot/indexer-worker user1@domain.com Sent - 5500/7517 [0x401d19] Jan 4 15:17:17 bubba dovecot: master: Error: service(indexer-worker): child 21821 returned error 83 (Out of memory (vsz_limit=256 MB, you may need to increase it)) Jan 4 15:17:17 bubba dovecot: indexer: Error: Indexer worker disconnected, discarding 1 requests for user1@domain.com Jan 4 15:30:48 bubba dovecot: imap(user2@domain.com): Error: Cached message size smaller than expected (822 < 1493) Jan 4 15:30:48 bubba dovecot: imap(user2@domain.com): Error: Corrupted index cache file /var/mail/amfes.com/lmiller/mdbox/mailboxes/Sent/dbox-Mails/dovecot.index.cache: Broken physical size for mail UID 1786 Jan 4 15:30:48 bubba dovecot: imap(user2@domain.com): Error: read(/var/mail/amfes.com/lmiller/mdbox/storage/m.208) failed: Input/output error (FETCH for mailbox Sent UID 1786) Jan 4 16:17:20 bubba dovecot: indexer-worker(user1@domain.com): Fatal:
Jan 4 16:17:20 bubba dovecot: indexer-worker(user1@domain.com): Error: Raw backtrace: /usr/local/lib/dovecot/libdovecot.so.0(+0x3ed0a) [0x7ffc91276d0a] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x3ed56) [0x7ffc91276d56] -> /usr/local/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7ffc9124fd08] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x4f310) [0x7ffc91287310] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x3b965) [0x7ffc91273965] -> /usr/local/lib/dovecot/libdovecot.so.0(buffer_write+0x7c) [0x7ffc912740ec] -> /usr/local/lib/dovecot/lib21_fts_solr_plugin.so(+0x3292) [0x7ffc8fa6e292] -> /usr/local/lib/dovecot/lib21_fts_solr_plugin.so(+0x3a97) [0x7ffc8fa6ea97] -> /usr/local/lib/dovecot/lib20_fts_plugin.so(fts_backend_update_set_build_key+0x2c) [0x7ffc8fc7babc] -> /usr/local/lib/dovecot/lib20_fts_plugin.so(fts_build_mail+0x2d1) [0x7ffc8fc7c561] -> /usr/local/lib/dovecot/lib20_fts_plugin.so(+0xc630) [0x7ffc8fc81630] -> dovecot/indexer-worker user1@domain.com Sent - 5500/7517 [0x40245f] -> dovecot/indexer-worker [user1@domain.com Sent
- 5500/7517]() [0x4027dd] -> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x36) [0x7ffc912830f6] -> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x9f) [0x7ffc9128417f] -> /usr/local/lib/dovecot/libdovecot.so.0(io_loop_run+0x28) [0x7ffc91283098] -> /usr/local/lib/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7ffc91270123] -> dovecot/indexer-worker user1@domain.com Sent - 5500/7517 [0x401f29] -> /lib/libc.so.6(__libc_start_main+0xfe) [0x7ffc90ed3d8e] -> dovecot/indexer-worker user1@domain.com Sent - 5500/7517 [0x401d19] Jan 4 16:17:20 bubba dovecot: master: Error: service(indexer-worker): child 22927 returned error 83 (Out of memory (vsz_limit=256 MB, you may need to increase it)) Jan 4 16:17:20 bubba dovecot: indexer: Error: Indexer worker disconnected, discarding 1 requests for user1@domain.com
-- Daniel
On 5.1.2012, at 2.24, Daniel L. Miller wrote:
I thought I had cleared out the corruption I had before - perhaps I was mistaken. What steps should I take to help locate these issues? Currently using 2.1rc1. I see the following errors in my logs, including out of memory and message size issues (at 15:30): .. Jan 4 05:17:17 bubba dovecot: master: Error: service(indexer-worker): child 10896 returned error 83 (Out of memory (vsz_limit=256 MB, you may need to increase it)) Jan 4 06:17:17 bubba dovecot: indexer-worker(user1@domain.com): Fatal: pool_system_realloc(134217728): Out of memory
The problem is clearly that index-worker's vsz_limit is too low. Increase it (or default_vsz_limit).
On 6.1.2012, at 12.55, Timo Sirainen wrote:
Jan 4 05:17:17 bubba dovecot: master: Error: service(indexer-worker): child 10896 returned error 83 (Out of memory (vsz_limit=256 MB, you may need to increase it)) Jan 4 06:17:17 bubba dovecot: indexer-worker(user1@domain.com): Fatal: pool_system_realloc(134217728): Out of memory
The problem is clearly that index-worker's vsz_limit is too low. Increase it (or default_vsz_limit).
Although the source of the out-of-memory
/usr/local/lib/dovecot/libdovecot.so.0(buffer_write+0x7c) [0x7f0ec1a550ec] -> /usr/local/lib/dovecot/lib21_fts_solr_plugin.so(+0x3292) [0x7f0ec024f292] ->
is something that shouldn't really be happening. I guess the Solr plugin wastes memory unnecessarily, I'll see what I can do about it. But for now just increase vsz limit.
On Fri, 2012-01-06 at 12:57 +0200, Timo Sirainen wrote:
On 6.1.2012, at 12.55, Timo Sirainen wrote:
Jan 4 05:17:17 bubba dovecot: master: Error: service(indexer-worker): child 10896 returned error 83 (Out of memory (vsz_limit=256 MB, you may need to increase it)) Jan 4 06:17:17 bubba dovecot: indexer-worker(user1@domain.com): Fatal: pool_system_realloc(134217728): Out of memory
The problem is clearly that index-worker's vsz_limit is too low. Increase it (or default_vsz_limit).
Although the source of the out-of-memory
/usr/local/lib/dovecot/libdovecot.so.0(buffer_write+0x7c) [0x7f0ec1a550ec] -> /usr/local/lib/dovecot/lib21_fts_solr_plugin.so(+0x3292) [0x7f0ec024f292] ->
is something that shouldn't really be happening. I guess the Solr plugin wastes memory unnecessarily, I'll see what I can do about it. But for now just increase vsz limit.
I don't see any obvious reason why it would be using a lot of memory, unless you have a message that has huge (MIME) headers. See if http://hg.dovecot.org/dovecot-2.1/rev/380b0667e0a5 helps / logs a warning about it.
On 1/6/2012 2:57 AM, Timo Sirainen wrote:
On 6.1.2012, at 12.55, Timo Sirainen wrote:
Jan 4 05:17:17 bubba dovecot: master: Error: service(indexer-worker): child 10896 returned error 83 (Out of memory (vsz_limit=256 MB, you may need to increase it)) Jan 4 06:17:17 bubba dovecot: indexer-worker(user1@domain.com): Fatal: pool_system_realloc(134217728): Out of memory The problem is clearly that index-worker's vsz_limit is too low. Increase it (or default_vsz_limit). Although the source of the out-of-memory
/usr/local/lib/dovecot/libdovecot.so.0(buffer_write+0x7c) [0x7f0ec1a550ec] -> /usr/local/lib/dovecot/lib21_fts_solr_plugin.so(+0x3292) [0x7f0ec024f292] ->
is something that shouldn't really be happening. I guess the Solr plugin wastes memory unnecessarily, I'll see what I can do about it. But for now just increase vsz limit.
I set default_vsz_limit = 1024M. Those errors appear gone - but I do have messages like:
Jan 6 09:22:42 bubba dovecot: indexer-worker(user1@domain.com): Error:
fts_solr: Indexing failed: 400 Illegal character ((CTRL-CHAR, code 18))
at [row,col {unknown-source}]: [482765,16]
Jan 6 09:22:42 bubba dovecot: indexer-worker: Error:
Google seems to indicate that Solr cannot handle "invalid" characters - and that it is the responsibility of the calling program to strip out such. A quick search shows me a both an individual character comparison in Java and a regex used for the purpose. Is there any "illegal character protection" in the Dovecot Solr plugin?
-- Daniel
On 6.1.2012, at 19.30, Daniel L. Miller wrote:
Jan 6 09:22:42 bubba dovecot: indexer-worker(user1@domain.com): Error: fts_solr: Indexing failed: 400 Illegal character ((CTRL-CHAR, code 18)) at [row,col {unknown-source}]: [482765,16] Jan 6 09:22:42 bubba dovecot: indexer-worker: Error:
Google seems to indicate that Solr cannot handle "invalid" characters - and that it is the responsibility of the calling program to strip out such. A quick search shows me a both an individual character comparison in Java and a regex used for the purpose. Is there any "illegal character protection" in the Dovecot Solr plugin?
Yes, there is. So I'm not really sure what it's complaining about. Are you using the "solr" or "solr_old" backend?
On 1/6/2012 9:36 AM, Timo Sirainen wrote:
On 6.1.2012, at 19.30, Daniel L. Miller wrote:
Jan 6 09:22:42 bubba dovecot: indexer-worker(user1@domain.com): Error: fts_solr: Indexing failed: 400 Illegal character ((CTRL-CHAR, code 18)) at [row,col {unknown-source}]: [482765,16] Jan 6 09:22:42 bubba dovecot: indexer-worker: Error:
Google seems to indicate that Solr cannot handle "invalid" characters - and that it is the responsibility of the calling program to strip out such. A quick search shows me a both an individual character comparison in Java and a regex used for the purpose. Is there any "illegal character protection" in the Dovecot Solr plugin? Yes, there is. So I'm not really sure what it's complaining about. Are you using the "solr" or "solr_old" backend?
"Solr".
plugin { fts = solr fts_solr = url=http://localhost:8983/solr/ }
-- Daniel
participants (2)
-
Daniel L. Miller
-
Timo Sirainen