Frequent Out of Memory for service(config)
Hello Group,
We have dovecot deployed as solely a Pop3 service that is used by our applications to pass mail from one application to another internally. We have roughly 4 applications that connect to the Pop3 service every 2 seconds, to check for new messages and pop them for processing if they are present. Depending on the site, we have between 1024-2048MB of memory set for default_vsz_limit. In all systems we see the Out of memory alert several times a day. We previously did not see this at all when running on CentOS6, with less memory.
We have tried increasing the memory to the vsz_limit up to 2gb without success.
We are running on CentOS 7 servers, running dovecot 2.3.6 (7eab80676) (from the dovecot repo).
Can anyone advise any other settings that could be modified in order to correct these out of memory issues?
# dovecot -n # 2.3.6 (7eab80676): /etc/dovecot/dovecot.conf # OS: Linux 3.10.0-957.5.1.el7.x86_64 x86_64 CentOS Linux release 7.6.1810 (Core) # Hostname: ****** #<modified for security> auth_cache_size = 10 M auth_verbose = yes default_vsz_limit = 1 G instance_name = Pop3 Mail Service listen = 10.*.*.* #<modified for security> log_path = /var/log/dovecot.log login_greeting = Pop3 Mail Service login_trusted_networks = 10.*.*.* 10.*.*.* 10.*.*.* 10.*.*.* 10.*.*.* #<modified for security> mail_location = maildir:~/Maildir namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { args = cache_key=#hidden_use-P_to_show# driver = pam } protocols = pop3 ssl_cert =
May 10 06:44:05 config: Fatal: pool_system_malloc(8192): Out of memory May 10 06:44:05 config: Fatal: master: service(config): child 27887 returned error 83 (Out of memory (service config { vsz_limit=1024 MB }, you may need to increase it) - set CORE_OUTOFMEM=1 environment to get core dump) May 10 14:56:31 config: Fatal: pool_system_malloc(8192): Out of memory May 10 14:56:31 config: Fatal: master: service(config): child 2495 returned error 83 (Out of memory (service config { vsz_limit=1024 MB }, you may need to increase it) - set CORE_OUTOFMEM=1 environment to get core dump) May 10 23:09:02 config: Fatal: pool_system_malloc(8192): Out of memory May 10 23:09:02 config: Fatal: master: service(config): child 10581 returned error 83 (Out of memory (service config { vsz_limit=1024 MB }, you may need to increase it) - set CORE_OUTOFMEM=1 environment to get core dump) May 11 10:18:43 config: Fatal: pool_system_malloc(8192): Out of memory May 11 10:18:43 config: Fatal: master: service(config): child 30511 returned error 83 (Out of memory (service config { vsz_limit=1024 MB }, you may need to increase it) - set CORE_OUTOFMEM=1 environment to get core dump) May 11 18:30:59 config: Fatal: pool_system_malloc(1608): Out of memory May 11 18:30:59 config: Fatal: master: service(config): child 12134 returned error 83 (Out of memory (service config { vsz_limit=1024 MB }, you may need to increase it) - set CORE_OUTOFMEM=1 environment to get core dump) May 12 02:57:28 config: Fatal: pool_system_malloc(8192): Out of memory May 12 02:57:28 config: Fatal: master: service(config): child 24940 returned error 83 (Out of memory (service config { vsz_limit=1024 MB }, you may need to increase it) - set CORE_OUTOFMEM=1 environment to get core dump) May 12 11:07:34 config: Fatal: pool_system_malloc(8192): Out of memory May 12 11:07:35 config: Fatal: master: service(config): child 8203 returned error 83 (Out of memory (service config { vsz_limit=1024 MB }, you may need to increase it) - set CORE_OUTOFMEM=1 environment to get core dump) May 12 19:17:52 config: Fatal: pool_system_malloc(8192): Out of memory May 12 19:17:52 config: Fatal: master: service(config): child 14623 returned error 83 (Out of memory (service config { vsz_limit=1024 MB }, you may need to increase it) - set CORE_OUTOFMEM=1 environment to get core dump) May 13 03:26:34 config: Fatal: pool_system_malloc(8192): Out of memory May 13 03:26:35 config: Fatal: master: service(config): child 21479 returned error 83 (Out of memory (service config { vsz_limit=1024 MB }, you may need to increase it) - set CORE_OUTOFMEM=1 environment to get core dump) May 13 11:35:42 config: Fatal: pool_system_malloc(8192): Out of memory May 13 11:35:43 config: Fatal: master: service(config): child 26900 returned error 83 (Out of memory (service config { vsz_limit=1024 MB }, you may need to increase it) - set CORE_OUTOFMEM=1 environment to get core dump) May 13 19:34:26 config: Fatal: pool_system_malloc(8192): Out of memory May 13 19:34:26 config: Fatal: master: service(config): child 32187 returned error 83 (Out of memory (service config { vsz_limit=1024 MB }, you may need to increase it) - set CORE_OUTOFMEM=1 environment to get core dump).
Thanks,
Kevin
On 13.5.2019 22.56, Root Kev via dovecot wrote:
Hello Group,
We have dovecot deployed as solely a Pop3 service that is used by our applications to pass mail from one application to another internally. We have roughly 4 applications that connect to the Pop3 service every 2 seconds, to check for new messages and pop them for processing if they are present. Depending on the site, we have between 1024-2048MB of memory set for default_vsz_limit. In all systems we see the Out of memory alert several times a day. We previously did not see this at all when running on CentOS6, with less memory.
We have tried increasing the memory to the vsz_limit up to 2gb without success.
We are running on CentOS 7 servers, running dovecot 2.3.6 (7eab80676) (from the dovecot repo).
Can anyone advise any other settings that could be modified in order to correct these out of memory issues?
# dovecot -n # 2.3.6 (7eab80676): /etc/dovecot/dovecot.conf # OS: Linux 3.10.0-957.5.1.el7.x86_64 x86_64 CentOS Linux release 7.6.1810 (Core) # Hostname: ****** #<modified for security> auth_cache_size = 10 M auth_verbose = yes default_vsz_limit = 1 G instance_name = Pop3 Mail Service listen = 10.*.*.* #<modified for security> log_path = /var/log/dovecot.log login_greeting = Pop3 Mail Service login_trusted_networks = 10.*.*.* 10.*.*.* 10.*.*.* 10.*.*.* 10.*.*.* #<modified for security> mail_location = maildir:~/Maildir namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { args = cache_key=#hidden_use-P_to_show# driver = pam } protocols = pop3 ssl_cert =
May 10 06:44:05 config: Fatal: pool_system_malloc(8192): Out of memory May 10 06:44:05 config: Fatal: master: service(config): child 27887 returned error 83 (Out of memory (service config { vsz_limit=1024 MB }, you may need to increase it) - set CORE_OUTOFMEM=1 environment to get core dump)
Can you try setting
import_environment = $import_environment CORE_OUTOFMEM=1
and see if it causes coredump?
Aki
Hi Aki,
I put in the line that you recommended as well as the " echo 'DAEMON_COREFILE_LIMIT="unlimited"' >> /etc/sysconfig/dovecot". Unfortunately, the logs are saying that core dumps are still disabled.
May 17 13:30:17 config: Fatal: pool_system_malloc(8192): Out of memory May 17 13:30:17 pop3(emx-echoworx): Error: Error reading configuration: read(/var/run/dovecot/config) failed: EOF May 17 13:30:17 config: Fatal: master: service(config): child 11530 killed with signal 6 (core dumps disabled - https://dovecot.org/bugreport.html#coredumps)
On Wed, May 15, 2019 at 4:34 AM Aki Tuomi aki.tuomi@open-xchange.com wrote:
On 13.5.2019 22.56, Root Kev via dovecot wrote:
Hello Group,
We have dovecot deployed as solely a Pop3 service that is used by our applications to pass mail from one application to another internally. We have roughly 4 applications that connect to the Pop3 service every 2 seconds, to check for new messages and pop them for processing if they are present. Depending on the site, we have between 1024-2048MB of memory set for default_vsz_limit. In all systems we see the Out of memory alert several times a day. We previously did not see this at all when running on CentOS6, with less memory.
We have tried increasing the memory to the vsz_limit up to 2gb without success.
We are running on CentOS 7 servers, running dovecot 2.3.6 (7eab80676) (from the dovecot repo).
Can anyone advise any other settings that could be modified in order to correct these out of memory issues?
# dovecot -n # 2.3.6 (7eab80676): /etc/dovecot/dovecot.conf # OS: Linux 3.10.0-957.5.1.el7.x86_64 x86_64 CentOS Linux release 7.6.1810 (Core) # Hostname: ****** #<modified for security> auth_cache_size = 10 M auth_verbose = yes default_vsz_limit = 1 G instance_name = Pop3 Mail Service listen = 10.*.*.* #<modified for security> log_path = /var/log/dovecot.log login_greeting = Pop3 Mail Service login_trusted_networks = 10.*.*.* 10.*.*.* 10.*.*.* 10.*.*.* 10.*.*.* #<modified for security> mail_location = maildir:~/Maildir namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { args = cache_key=#hidden_use-P_to_show# driver = pam } protocols = pop3 ssl_cert =
May 10 06:44:05 config: Fatal: pool_system_malloc(8192): Out of memory May 10 06:44:05 config: Fatal: master: service(config): child 27887 returned error 83 (Out of memory (service config { vsz_limit=1024 MB }, you may need to increase it) - set CORE_OUTOFMEM=1 environment to get core dump)
Can you try setting
import_environment = $import_environment CORE_OUTOFMEM=1
and see if it causes coredump?
Aki
Hello Aki + group.
The following is the latest dump output that I have. Does this give any indication of what is wrong? I am not sure what this is really showing me...
# coredumpctl gdb -1 PID: 14432 (config) UID: 0 (root) GID: 0 (root) Signal: 6 (ABRT) Timestamp: Wed 2019-06-05 13:25:32 UTC (4h 52min ago) Command Line: dovecot-Pop3 Mail Service/config Executable: /usr/libexec/dovecot/config Control Group: /system.slice/dovecot.service Unit: dovecot.service Slice: system.slice Boot ID: 9ae422871a814d699f5feb4ca52d3b69 Machine ID: 05cb8c7b39fe0f70e3ce97e5beab809d Hostname: <*****REMOVED******> Coredump: /var/lib/systemd/coredump/core.config.0.9ae422871a814d699f5feb4ca52d3b69.14432.1559741132000000.xz Message: Process 14432 (config) of user 0 dumped core.
Stack trace of thread 14432:
#0 0x00007fd25a809207 raise (libc.so.6)
#1 0x00007fd25a80a8f8 abort (libc.so.6)
#2 0x00007fd25ac79567 fatal_handler_real (libdovecot.so.0)
#3 0x00007fd25ac79651 i_internal_fatal_handler(libdovecot.so.0) #4 0x00007fd25abdf2d9 i_fatal_status (libdovecot.so.0) #5 0x00007fd25ac9b900 pool_system_malloc (libdovecot.so.0) #6 0x00007fd25aca220a o_stream_grow_buffer (libdovecot.so.0) #7 0x00007fd25aca2506 o_stream_add (libdovecot.so.0) #8 0x00007fd25aca3388 o_stream_file_sendv (libdovecot.so.0) #9 0x00007fd25aca03f5 o_stream_sendv_int (libdovecot.so.0) #10 0x00007fd25aca0a5c o_stream_nsendv (libdovecot.so.0) #11 0x00007fd25aca0aca o_stream_nsend (libdovecot.so.0) #12 0x0000556e753617dc config_connection_input (config) #13 0x00007fd25ac91e0f io_loop_call_io (libdovecot.so.0) #14 0x00007fd25ac9384b io_loop_handler_run_internal (libdovecot.so.0) #15 0x00007fd25ac91f16 io_loop_handler_run (libdovecot.so.0) #16 0x00007fd25ac92138 io_loop_run (libdovecot.so.0) #17 0x00007fd25ac07973 master_service_run (libdovecot.so.0) #18 0x0000556e7535edc9 main (config) #19 0x00007fd25a7f53d5 __libc_start_main (libc.so.6) #20 0x0000556e7535ee3b _start (config)
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-114.el7 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: http://www.gnu.org/software/gdb/bugs/... Reading symbols from /usr/libexec/dovecot/config...Reading symbols from /usr/lib/debug/usr/libexec/dovecot/config.debug...done. done. [New LWP 14432] Core was generated by `dovecot-Pop3 Mail Service/config'. Program terminated with signal 6, Aborted. #0 0x00007fd25a809207 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55 55 return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);
Thanks,
Kevin
On Wed, May 15, 2019 at 4:34 AM Aki Tuomi aki.tuomi@open-xchange.com wrote:
On 13.5.2019 22.56, Root Kev via dovecot wrote:
Hello Group,
We have dovecot deployed as solely a Pop3 service that is used by our applications to pass mail from one application to another internally. We have roughly 4 applications that connect to the Pop3 service every 2 seconds, to check for new messages and pop them for processing if they are present. Depending on the site, we have between 1024-2048MB of memory set for default_vsz_limit. In all systems we see the Out of memory alert several times a day. We previously did not see this at all when running on CentOS6, with less memory.
We have tried increasing the memory to the vsz_limit up to 2gb without success.
We are running on CentOS 7 servers, running dovecot 2.3.6 (7eab80676) (from the dovecot repo).
Can anyone advise any other settings that could be modified in order to correct these out of memory issues?
# dovecot -n # 2.3.6 (7eab80676): /etc/dovecot/dovecot.conf # OS: Linux 3.10.0-957.5.1.el7.x86_64 x86_64 CentOS Linux release 7.6.1810 (Core) # Hostname: ****** #<modified for security> auth_cache_size = 10 M auth_verbose = yes default_vsz_limit = 1 G instance_name = Pop3 Mail Service listen = 10.*.*.* #<modified for security> log_path = /var/log/dovecot.log login_greeting = Pop3 Mail Service login_trusted_networks = 10.*.*.* 10.*.*.* 10.*.*.* 10.*.*.* 10.*.*.* #<modified for security> mail_location = maildir:~/Maildir namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { args = cache_key=#hidden_use-P_to_show# driver = pam } protocols = pop3 ssl_cert =
May 10 06:44:05 config: Fatal: pool_system_malloc(8192): Out of memory May 10 06:44:05 config: Fatal: master: service(config): child 27887 returned error 83 (Out of memory (service config { vsz_limit=1024 MB }, you may need to increase it) - set CORE_OUTOFMEM=1 environment to get core dump)
Can you try setting
import_environment = $import_environment CORE_OUTOFMEM=1
and see if it causes coredump?
Aki
On 2019-05-13 21:56, Root Kev via dovecot wrote:
Hello Group,
We have dovecot deployed as solely a Pop3 service that is used by our applications to pass mail from one application to another internally.
We have roughly 4 applications that connect to the Pop3 service every 2 seconds, to check for new messages and pop them for processing if they are present. Depending on the site, we have between 1024-2048MB of memory set for default_vsz_limit. In all systems we see the Out of memory alert several times a day. We previously did not see this at all when running on CentOS6, with less memory.
I see this too on servers running quota-service (dunno if it is related).
Tom
Hey Tom,
We don't run any quota-service in dovecot (I just verified just in case as well). Thanks for the suggestion though.
Kevin
On Sun, May 19, 2019 at 9:21 AM Tom Sommer via dovecot dovecot@dovecot.org wrote:
On 2019-05-13 21:56, Root Kev via dovecot wrote:
Hello Group,
We have dovecot deployed as solely a Pop3 service that is used by our applications to pass mail from one application to another internally. We have roughly 4 applications that connect to the Pop3 service every 2 seconds, to check for new messages and pop them for processing if they are present. Depending on the site, we have between 1024-2048MB of memory set for default_vsz_limit. In all systems we see the Out of memory alert several times a day. We previously did not see this at all when running on CentOS6, with less memory.
I see this too on servers running quota-service (dunno if it is related).
Tom
On 13 May 2019, at 22.56, Root Kev via dovecot dovecot@dovecot.org wrote:
May 13 11:35:43 config: Fatal: master: service(config): child 26900 returned error 83 (Out of memory (service config { vsz_limit=1024 MB }, you may need to increase it) - set CORE_OUTOFMEM=1 environment to get core dump)
If you set ssl_dh setting this most likely gets fixed.
participants (4)
-
Aki Tuomi
-
Root Kev
-
Timo Sirainen
-
Tom Sommer