-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512

Hi all,

we're experiencing "Out of Memory" Problems caused by Sieve-Filters delivering mail to huge IMAP folders. From the logs, I'm beginning to think the problem is that the sieve/lmtp process loads the "dovecot.index.cache" file apparently twice into memory to do something with it.

We've set our memory limit per process to 256MB - the cache file of the folders we're having issues with is exactly 128MB. Mails coming in that should be filtered to these folders are being denied, an error is being logged.

Apparently, there would be two easy solutions to this: increasing memory OR decreasing folder size.

Both are not possible. Increasing memory just means we will have this issue again in some weeks. Deacreasing mailbox size isn't possible since the folders contain archived system mails from our backend software which need to be kept for a while.

Currently, we're creating a new folder each time we reach this limit, but that isn't exactly the best way to go for the future. In the end, the folders are for archiving purposes and therefore should not be managed at all since all this is automated.

I hope there's some obvious switch I've just not seen in the config. I actually can't imagine that this is default or wanted behaviour since there must be people with large inboxes and sieve filters that either also have this problem or don't have it since they fixed it.

The problem clearly is reproducable. Each time the cache file hits 128MB - which happens at around 200k mails in the folder - (with our current setting of 256MB memory limit per process), things go down and our mailqueues reach 10k undelivered mails within a few hours.

I hope you can help me

Sincerly Fabian

OS

Linux 2.6.32-16-pve #1 SMP Mon Oct 22 08:38:13 CEST 2012 i686 GNU/Linux

(The IMAP server is running within a debian squeeze openVZ container on the Proxmox Virtual Environment Kernel; filesystem for the containers is ext3)

VERSION

2.1.14 (bf80034a547d)

CONFIG

# 2.1.14 (bf80034a547d): /etc/dovecot/dovecot.conf # OS: Linux 2.6.32-16-pve i686 Debian 6.0.7 simfs auth_cache_size = 10 M mail_debug = yes mail_location = maildir:/var/mail/%u:LAYOUT=fs mail_max_userip_connections = 10000 maildir_very_dirty_syncs = yes managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave namespace inbox { inbox = yes location = mailbox Archive { auto = no special_use = \Archive } mailbox Archives { auto = subscribe special_use = \Archive } mailbox Drafts { auto = subscribe special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { auto = subscribe special_use = \Sent } mailbox Trash { auto = subscribe special_use = \Trash } prefix = separator = / } passdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql } plugin { sieve = /var/mail/%u/.dovecot.sieve sieve_dir = /var/mail/%u/sieve } protocols = imap lmtp sieve service imap-login { client_limit = 5 process_limit = 0 process_min_avail = 4 service_count = 0 } service imap { client_limit = 10 process_limit = 1024 process_min_avail = 4 } service lmtp { client_limit = 20 inet_listener lmtp { address = *, [::] port = 24 } process_limit = 0 process_min_avail = 10 vsz_limit = 256 M } service managesieve-login { inet_listener sieve { port = 4190 } } service managesieve { client_limit = 10 process_limit = 0 process_min_avail = 4 } ssl = required ssl_ca = </etc/ssl/certs/wildcard.hexonet.net.ca ssl_cert = </etc/ssl/certs/wildcard.hexonet.net.crt ssl_key = </etc/ssl/certs/wildcard.hexonet.net.key userdb { args = uid=vmail gid=vmail home=/var/mail/%u driver = static } protocol lmtp { mail_plugins = sieve }

LOG

May 12 06:36:36: Debug: none: root=, index=, control=, inbox=, alt= May 12 06:36:36: Debug: auth input: USERNAME@hexonet.net uid=5000 gid=5000 home=/var/mail/USERNAME@hexonet.net May 12 06:36:36: Debug: Effective uid=5000, gid=5000, home=/var/mail/USERNAME@hexonet.net May 12 06:36:36: Debug: Namespace inbox: type=private, prefix=, sep=/, inbox=yes, hidden=no, list=yes, subscriptions=yes location=maildir:/var/mail/USERNAME@hexonet.net:LAYOUT=fs May 12 06:36:36: Debug: fs: root=/var/mail/USERNAME@hexonet.net, index=, control=, inbox=/var/mail/USERNAME@hexonet.net, alt= May 12 06:36:36: Debug: sieve: include: sieve_global_dir is not set; it is currently not possible to include :global' scripts. May 12 06:36:36: Debug: PJyQMOZrcFN7awAA9jyUPQ: sieve: using the following location for user's Sieve script: /var/mail/USERNAME@hexonet.net/.dovecot.sieve May 12 06:36:36: Debug: PJyQMOZrcFN7awAA9jyUPQ: sieve: opening script /var/mail/USERNAME@hexonet.net/.dovecot.sieve May 12 06:36:36: Debug: PJyQMOZrcFN7awAA9jyUPQ: sieve: script binary /var/mail/USERNAME@hexonet.net/.dovecot.svbin successfully loaded May 12 06:36:36: Debug: PJyQMOZrcFN7awAA9jyUPQ: sieve: binary save: not saving binary /var/mail/USERNAME@hexonet.net/.dovecot.svbin, because it is already stored May 12 06:36:36: Debug: PJyQMOZrcFN7awAA9jyUPQ: sieve: executing script from /var/mail/USERNAME@hexonet.net/.dovecot.svbin May 12 06:36:36: PJyQMOZrcFN7awAA9jyUPQ: sieve: msgid=unspecified: stored mail into mailbox 'IMAP-FOLDER' May 12 06:36:41 Debug: none: root=, index=, control=, inbox=, alt= May 12 06:36:41 Debug: auth input: USERNAME@hexonet.net uid=5000 gid=5000 home=/var/mail/USERNAME@hexonet.net May 12 06:36:41: Debug: Effective uid=5000, gid=5000, home=/var/mail/USERNAME@hexonet.net May 12 06:36:41: Debug: Namespace inbox: type=private, prefix=, sep=/, inbox=yes, hidden=no, list=yes, subscriptions=yes location=maildir:/var/mail/USERNAME@hexonet.net:LAYOUT=fs May 12 06:36:41: Debug: fs: root=/var/mail/USERNAME@hexonet.net, index=, control=, inbox=/var/mail/USERNAME@hexonet.net, alt= May 12 06:36:41: Debug: sieve: include: sieve_global_dir is not set; it is currently not possible to include :global' scripts. May 12 06:36:41: Debug: QJyQMOZrcFN7awAA9jyUPQ: sieve: using the following location for user's Sieve script: /var/mail/USERNAME@hexonet.net/.dovecot.sieve May 12 06:36:41: Debug: QJyQMOZrcFN7awAA9jyUPQ: sieve: opening script /var/mail/USERNAME@hexonet.net/.dovecot.sieve May 12 06:36:41: Debug: QJyQMOZrcFN7awAA9jyUPQ: sieve: script binary /var/mail/USERNAME@hexonet.net/.dovecot.svbin successfully loaded May 12 06:36:41: Debug: QJyQMOZrcFN7awAA9jyUPQ: sieve: binary save: not saving binary /var/mail/USERNAME@hexonet.net/.dovecot.svbin, because it is already stored May 12 06:36:41: Debug: QJyQMOZrcFN7awAA9jyUPQ: sieve: executing script from /var/mail/USERNAME@hexonet.net/.dovecot.svbin May 12 06:36:41: Fatal: pool_system_realloc(268435456): Out of memory May 12 06:36:41: Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x45951) [0x219951] -> /usr/lib/dovecot/libdovecot.so.0(+0x459bf) [0x2199bf] -> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x1e8dc5] -> /usr/lib/dovecot/libdovecot.so.0(+0x5806e) [0x22c06e] -> /usr/lib/dovecot/libdovecot.so.0(+0x40ec8) [0x214ec8] -> /usr/lib/dovecot/libdovecot.so.0(buffer_get_space_unsafe+0x85) [0x2153b5] -> /usr/lib/dovecot/libdovecot.so.0(buffer_append_space_unsafe+0x2b) [0x2154cb] -> /usr/lib/dovecot/libdovecot-storage.so.0(mail_cache_map+0x219) [0x506cf9] -> /usr/lib/dovecot/libdovecot-storage.so.0(mail_cache_open_and_verify+0x75) [0x507245] -> /usr/lib/dovecot/libdovecot-storage.so.0(mail_cache_register_get_list+0x50) [0x5092a0] -> /usr/lib/dovecot/libdovecot-storage.so.0(index_mail_parse_header_init+0x15f) [0x4f6def] -> /usr/lib/dovecot/libdovecot-storage.so.0(index_mail_cache_parse_init+0x7c) [0x4f73fc] -> /usr/lib/dovecot/libdovecot-storage.so.0(maildir_save_add+0x1d7) [0x4a0a87] -> /usr/lib/dovecot/libdovecot-storage.so.0(maildir_save_begin+0x1d3) [0x4a0dd3] -> /usr/lib/dovecot/libdovecot-storage.so.0(mailbox_save_begin+0x58) [0x4d7988] -> /usr/lib/dovecot/libdovecot-storage.so.0(mail_storage_copy+0xba) [0x4d163a] -> /usr/lib/dovecot/libdovecot-storage.so.0(maildir_copy+0x4b) [0x49bacb]

• -> /usr/lib/dovecot/libdovecot-storage.so.0(mailbox_copy+0x54) [0x4d78b4] -> /usr/lib/dovecot/libdovecot-sieve.so.0(+0x35f5f) [0xc52f5f] -> /usr/lib/dovecot/libdovecot-sieve.so.0(sieve_result_execute+0x16b) [0xc49efb] -> /usr/lib/dovecot/libdovecot-sieve.so.0(sieve_execute+0x69) [0xc5b139]
• -> /usr/lib/dovecot/modules/lib90_sieve_plugin.so(+0x2892) [0x75e892]
• -> /usr/lib/dovecot/libdovecot-lda.so.0(mail_deliver+0x4f) [0x19641f]
• -> dovecot/lmtp() [0x804c812] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x42) [0x227822] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0xd3) [0x2289e3]
• -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x40) [0x2277b0] -> /usr/lib/dovecot/libdovecot.so.0(master_service_run+0x2d) [0x21050d] May 12 06:36:41: Fatal: master: service(lmtp): child 27515 returned error 83 (Out of memory (service lmtp { vsz_limit=256 MB }, you may need to increase it))

Fabian Peter Systemadministrator HEXONET GmbH

T: +49 6841 69 84 308 F: +49 6841 69 84 199 E: fpeter@hexonet.net W: http://www.hexonet.net

HEXONET GmbH, Talstrasse 27, 66424 Homburg, Germany. CEO & General Manager: Jens Wagner, HRB 2839 (HOM), Amtsgericht Saarbrücken, VAT-ID: DE-138316882

HEXONET Services Inc., 104 - 7455 132nd Street, Surrey, B.C., V3W 1J8, Canada. CSO & General Manager: Robert Birkner

This email and any files transmitted are confidential and intended only or the person(s) directly addressed. If you are not the intended recipient, any use, copying, transmission, distribution, or other forms of dissemination is strictly prohibited. If you have received this email in error, please notify the sender immediately and permanently delete this email with any files that may be attached. -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.22 (Darwin) Comment: GPGTools - https://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBCgAGBQJTcHVYAAoJEPIZChNlZ+or5VQIAKegO6HwORlpEmP3YBkJnHXY sYsR1sP89Vm1Uzv4cVL30KF1P37NI9sZwHrO5H2aC5sqNSadaZ0lOUFncwlZwphh JsSS2Jye+56UW7XR92rUuwC8FPYRk6nOxTA0hHtAGW1IxCROz8P32V24cRlDE32k rcq2GT0V4laFlD5YNcqFWAClTqAholnHZCm/7VpBCevUTx0F+KVZxEth/xOf0ajY EmWoXat4Fihnly1YtdPqN5FqIyFdzBF8tyuWaUZN3BfO8QDso+tci/NRbe2yYkIK lhulr4PAvccsIqJ+u/9j+XN7SL9ZXDtU0w4xWQe+5SrGfFXJlXsuJoRgBdThn9I= =YP4k -----END PGP SIGNATURE-----