[Dovecot] ManageSieve OOM with Huge Folders
Fabian Peter
fpeter at hexonet.net
Mon May 12 07:16:40 UTC 2014
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
Hi all,
we're experiencing "Out of Memory" Problems caused by Sieve-Filters
delivering mail to huge IMAP folders. From the logs, I'm beginning to
think the problem is that the sieve/lmtp process loads the
"dovecot.index.cache" file apparently twice into memory to do
something with it.
We've set our memory limit per process to 256MB - the cache file of
the folders we're having issues with is exactly 128MB. Mails coming in
that should be filtered to these folders are being denied, an error is
being logged.
Apparently, there would be two easy solutions to this: increasing
memory OR decreasing folder size.
Both are not possible. Increasing memory just means we will have this
issue again in some weeks. Deacreasing mailbox size isn't possible
since the folders contain archived system mails from our backend
software which need to be kept for a while.
Currently, we're creating a new folder each time we reach this limit,
but that isn't exactly the best way to go for the future. In the end,
the folders are for archiving purposes and therefore should not be
managed at all since all this is automated.
I hope there's some obvious switch I've just not seen in the config. I
actually can't imagine that this is default or wanted behaviour since
there must be people with large inboxes and sieve filters that either
also have this problem or don't have it since they fixed it.
The problem clearly is reproducable. Each time the cache file hits
128MB - which happens at around 200k mails in the folder - (with our
current setting of 256MB memory limit per process), things go down and
our mailqueues reach 10k undelivered mails within a few hours.
I hope you can help me
Sincerly
Fabian
OS
- --
Linux 2.6.32-16-pve #1 SMP Mon Oct 22 08:38:13 CEST 2012 i686 GNU/Linux
(The IMAP server is running within a debian squeeze openVZ container
on the Proxmox Virtual Environment Kernel; filesystem for the
containers is ext3)
VERSION
- -------
2.1.14 (bf80034a547d)
CONFIG
- ------
# 2.1.14 (bf80034a547d): /etc/dovecot/dovecot.conf
# OS: Linux 2.6.32-16-pve i686 Debian 6.0.7 simfs
auth_cache_size = 10 M
mail_debug = yes
mail_location = maildir:/var/mail/%u:LAYOUT=fs
mail_max_userip_connections = 10000
maildir_very_dirty_syncs = yes
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope
encoded-character vacation subaddress comparator-i;ascii-numeric
relational regex imap4flags copy include variables body enotify
environment mailbox date ihave
namespace inbox {
inbox = yes
location =
mailbox Archive {
auto = no
special_use = \Archive
}
mailbox Archives {
auto = subscribe
special_use = \Archive
}
mailbox Drafts {
auto = subscribe
special_use = \Drafts
}
mailbox Junk {
special_use = \Junk
}
mailbox Sent {
auto = subscribe
special_use = \Sent
}
mailbox Trash {
auto = subscribe
special_use = \Trash
}
prefix =
separator = /
}
passdb {
args = /etc/dovecot/dovecot-sql.conf.ext
driver = sql
}
plugin {
sieve = /var/mail/%u/.dovecot.sieve
sieve_dir = /var/mail/%u/sieve
}
protocols = imap lmtp sieve
service imap-login {
client_limit = 5
process_limit = 0
process_min_avail = 4
service_count = 0
}
service imap {
client_limit = 10
process_limit = 1024
process_min_avail = 4
}
service lmtp {
client_limit = 20
inet_listener lmtp {
address = *, [::]
port = 24
}
process_limit = 0
process_min_avail = 10
vsz_limit = 256 M
}
service managesieve-login {
inet_listener sieve {
port = 4190
}
}
service managesieve {
client_limit = 10
process_limit = 0
process_min_avail = 4
}
ssl = required
ssl_ca = </etc/ssl/certs/wildcard.hexonet.net.ca
ssl_cert = </etc/ssl/certs/wildcard.hexonet.net.crt
ssl_key = </etc/ssl/certs/wildcard.hexonet.net.key
userdb {
args = uid=vmail gid=vmail home=/var/mail/%u
driver = static
}
protocol lmtp {
mail_plugins = sieve
}
LOG
- ---
May 12 06:36:36: Debug: none: root=, index=, control=, inbox=, alt=
May 12 06:36:36: Debug: auth input: USERNAME at hexonet.net uid=5000
gid=5000 home=/var/mail/USERNAME at hexonet.net
May 12 06:36:36: Debug: Effective uid=5000, gid=5000,
home=/var/mail/USERNAME at hexonet.net
May 12 06:36:36: Debug: Namespace inbox: type=private, prefix=, sep=/,
inbox=yes, hidden=no, list=yes, subscriptions=yes
location=maildir:/var/mail/USERNAME at hexonet.net:LAYOUT=fs
May 12 06:36:36: Debug: fs: root=/var/mail/USERNAME at hexonet.net,
index=, control=, inbox=/var/mail/USERNAME at hexonet.net, alt=
May 12 06:36:36: Debug: sieve: include: sieve_global_dir is not set;
it is currently not possible to include `:global' scripts.
May 12 06:36:36: Debug: PJyQMOZrcFN7awAA9jyUPQ: sieve: using the
following location for user's Sieve script:
/var/mail/USERNAME at hexonet.net/.dovecot.sieve
May 12 06:36:36: Debug: PJyQMOZrcFN7awAA9jyUPQ: sieve: opening script
/var/mail/USERNAME at hexonet.net/.dovecot.sieve
May 12 06:36:36: Debug: PJyQMOZrcFN7awAA9jyUPQ: sieve: script binary
/var/mail/USERNAME at hexonet.net/.dovecot.svbin successfully loaded
May 12 06:36:36: Debug: PJyQMOZrcFN7awAA9jyUPQ: sieve: binary save:
not saving binary /var/mail/USERNAME at hexonet.net/.dovecot.svbin,
because it is already stored
May 12 06:36:36: Debug: PJyQMOZrcFN7awAA9jyUPQ: sieve: executing
script from /var/mail/USERNAME at hexonet.net/.dovecot.svbin
May 12 06:36:36: PJyQMOZrcFN7awAA9jyUPQ: sieve: msgid=unspecified:
stored mail into mailbox 'IMAP-FOLDER'
May 12 06:36:41 Debug: none: root=, index=, control=, inbox=, alt=
May 12 06:36:41 Debug: auth input: USERNAME at hexonet.net uid=5000
gid=5000 home=/var/mail/USERNAME at hexonet.net
May 12 06:36:41: Debug: Effective uid=5000, gid=5000,
home=/var/mail/USERNAME at hexonet.net
May 12 06:36:41: Debug: Namespace inbox: type=private, prefix=, sep=/,
inbox=yes, hidden=no, list=yes, subscriptions=yes
location=maildir:/var/mail/USERNAME at hexonet.net:LAYOUT=fs
May 12 06:36:41: Debug: fs: root=/var/mail/USERNAME at hexonet.net,
index=, control=, inbox=/var/mail/USERNAME at hexonet.net, alt=
May 12 06:36:41: Debug: sieve: include: sieve_global_dir is not set;
it is currently not possible to include `:global' scripts.
May 12 06:36:41: Debug: QJyQMOZrcFN7awAA9jyUPQ: sieve: using the
following location for user's Sieve script:
/var/mail/USERNAME at hexonet.net/.dovecot.sieve
May 12 06:36:41: Debug: QJyQMOZrcFN7awAA9jyUPQ: sieve: opening script
/var/mail/USERNAME at hexonet.net/.dovecot.sieve
May 12 06:36:41: Debug: QJyQMOZrcFN7awAA9jyUPQ: sieve: script binary
/var/mail/USERNAME at hexonet.net/.dovecot.svbin successfully loaded
May 12 06:36:41: Debug: QJyQMOZrcFN7awAA9jyUPQ: sieve: binary save:
not saving binary /var/mail/USERNAME at hexonet.net/.dovecot.svbin,
because it is already stored
May 12 06:36:41: Debug: QJyQMOZrcFN7awAA9jyUPQ: sieve: executing
script from /var/mail/USERNAME at hexonet.net/.dovecot.svbin
May 12 06:36:41: Fatal: pool_system_realloc(268435456): Out of memory
May 12 06:36:41: Error: Raw backtrace:
/usr/lib/dovecot/libdovecot.so.0(+0x45951) [0x219951] ->
/usr/lib/dovecot/libdovecot.so.0(+0x459bf) [0x2199bf] ->
/usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x1e8dc5] ->
/usr/lib/dovecot/libdovecot.so.0(+0x5806e) [0x22c06e] ->
/usr/lib/dovecot/libdovecot.so.0(+0x40ec8) [0x214ec8] ->
/usr/lib/dovecot/libdovecot.so.0(buffer_get_space_unsafe+0x85)
[0x2153b5] ->
/usr/lib/dovecot/libdovecot.so.0(buffer_append_space_unsafe+0x2b)
[0x2154cb] ->
/usr/lib/dovecot/libdovecot-storage.so.0(mail_cache_map+0x219)
[0x506cf9] ->
/usr/lib/dovecot/libdovecot-storage.so.0(mail_cache_open_and_verify+0x75)
[0x507245] ->
/usr/lib/dovecot/libdovecot-storage.so.0(mail_cache_register_get_list+0x50)
[0x5092a0] ->
/usr/lib/dovecot/libdovecot-storage.so.0(index_mail_parse_header_init+0x15f)
[0x4f6def] ->
/usr/lib/dovecot/libdovecot-storage.so.0(index_mail_cache_parse_init+0x7c)
[0x4f73fc] ->
/usr/lib/dovecot/libdovecot-storage.so.0(maildir_save_add+0x1d7)
[0x4a0a87] ->
/usr/lib/dovecot/libdovecot-storage.so.0(maildir_save_begin+0x1d3)
[0x4a0dd3] ->
/usr/lib/dovecot/libdovecot-storage.so.0(mailbox_save_begin+0x58)
[0x4d7988] ->
/usr/lib/dovecot/libdovecot-storage.so.0(mail_storage_copy+0xba)
[0x4d163a] ->
/usr/lib/dovecot/libdovecot-storage.so.0(maildir_copy+0x4b) [0x49bacb]
- -> /usr/lib/dovecot/libdovecot-storage.so.0(mailbox_copy+0x54)
[0x4d78b4] -> /usr/lib/dovecot/libdovecot-sieve.so.0(+0x35f5f)
[0xc52f5f] ->
/usr/lib/dovecot/libdovecot-sieve.so.0(sieve_result_execute+0x16b)
[0xc49efb] ->
/usr/lib/dovecot/libdovecot-sieve.so.0(sieve_execute+0x69) [0xc5b139]
- -> /usr/lib/dovecot/modules/lib90_sieve_plugin.so(+0x2892) [0x75e892]
- -> /usr/lib/dovecot/libdovecot-lda.so.0(mail_deliver+0x4f) [0x19641f]
- -> dovecot/lmtp() [0x804c812] ->
/usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x42) [0x227822] ->
/usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0xd3) [0x2289e3]
- -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x40) [0x2277b0] ->
/usr/lib/dovecot/libdovecot.so.0(master_service_run+0x2d) [0x21050d]
May 12 06:36:41: Fatal: master: service(lmtp): child 27515 returned
error 83 (Out of memory (service lmtp { vsz_limit=256 MB }, you may
need to increase it))
- --
Fabian Peter
Systemadministrator
HEXONET GmbH
T: +49 6841 69 84 308
F: +49 6841 69 84 199
E: fpeter at hexonet.net
W: http://www.hexonet.net
HEXONET GmbH, Talstrasse 27, 66424 Homburg, Germany. CEO & General
Manager: Jens Wagner,
HRB 2839 (HOM), Amtsgericht Saarbrücken, VAT-ID: DE-138316882
HEXONET Services Inc., 104 - 7455 132nd Street, Surrey, B.C., V3W 1J8,
Canada. CSO &
General Manager: Robert Birkner
This email and any files transmitted are confidential and intended
only or the person(s)
directly addressed. If you are not the intended recipient, any use,
copying, transmission,
distribution, or other forms of dissemination is strictly prohibited.
If you have received
this email in error, please notify the sender immediately and
permanently delete this
email with any files that may be attached.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.22 (Darwin)
Comment: GPGTools - https://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQEcBAEBCgAGBQJTcHVYAAoJEPIZChNlZ+or5VQIAKegO6HwORlpEmP3YBkJnHXY
sYsR1sP89Vm1Uzv4cVL30KF1P37NI9sZwHrO5H2aC5sqNSadaZ0lOUFncwlZwphh
JsSS2Jye+56UW7XR92rUuwC8FPYRk6nOxTA0hHtAGW1IxCROz8P32V24cRlDE32k
rcq2GT0V4laFlD5YNcqFWAClTqAholnHZCm/7VpBCevUTx0F+KVZxEth/xOf0ajY
EmWoXat4Fihnly1YtdPqN5FqIyFdzBF8tyuWaUZN3BfO8QDso+tci/NRbe2yYkIK
lhulr4PAvccsIqJ+u/9j+XN7SL9ZXDtU0w4xWQe+5SrGfFXJlXsuJoRgBdThn9I=
=YP4k
-----END PGP SIGNATURE-----
More information about the dovecot
mailing list