I encountered this error on FreeBSD 9.3 with dovecot2-2.2.13_3
Short version; vsz_limit is 18,447 PB and it ran out of RAM. Suggestions for change?
Sep 6 03:39:32 mailjail dovecot: imap(dan): Panic: file imap-fetch.c: line 556 (imap_fetch_more): assertion failed: (ctx->client->output_cmd_lock == NULL || ctx->client->output_cmd_lock == cmd) Sep 6 03:39:32 mailjail dovecot: imap(dan): Fatal: master: service(imap): child 71153 killed with signal 6 (core not dumped - set service imap { drop_priv_before_exec=yes }) Sep 6 03:59:41 mailjail dovecot: imap(dan): Fatal: pool_system_realloc(2097152): Out of memory Sep 6 03:59:41 mailjail dovecot: imap(dan): Fatal: master: service(imap): child 67732 returned error 83 (Out of memory (service imap { vsz_limit=256 MB }, you may need to increase it) - set CORE_OUTOFMEM=1 environment to get core dump)
Background: I’m in the only user on this system, but this server is accessed by my phone, my laptop, my tablet, and perhaps a web interface.
Here are the non-default values:
$ doveconf -n
2.2.13: /usr/local/etc/dovecot/dovecot.conf
OS: FreeBSD 9.3-RELEASE amd64
auth_mechanisms = plain login first_valid_gid = 1001 first_valid_uid = 1001 mail_location = maildir:~/Maildir mail_max_userip_connections = 80 mail_privileged_group = mail passdb { args = scheme=SHA512-CRYPT /var/db/dovecot.users driver = passwd-file } protocols = imap service imap-login { inet_listener imap { address = 10.0.0.1 } inet_listener imaps { port = 0 } } ssl = required ssl_ca = </usr/local/etc/ssl/ca.pem ssl_cert = </usr/local/etc/ssl/server.pem ssl_key = </usr/local/etc/ssl/mailjail.example.org.nopassword.key userdb { args = /var/db/dovecot.users driver = passwd-file } verbose_proctitle = yes verbose_ssl = yes
But there are some interesting values when I look at the output of doveconf. Specifically, vsz_limit is 18,447 PB… yeah, that’s pretty big.
service imap-login { chroot = login client_limit = 0 drop_priv_before_exec = no executable = imap-login extra_groups = group = idle_kill = 0 inet_listener imap { address = 10.0.0.1 port = 143 reuse_port = no ssl = no } inet_listener imaps { address = port = 0 reuse_port = no ssl = yes } privileged_group = process_limit = 0 process_min_avail = 0 protocol = imap service_count = 1 type = login user = $default_login_user vsz_limit = 18446744073709551615 B }
On 6.9.2014 22:38, Dan Langille wrote:
I encountered this error on FreeBSD 9.3 with dovecot2-2.2.13_3
Short version; vsz_limit is 18,447 PB and it ran out of RAM. Suggestions for change?
Sep 6 03:39:32 mailjail dovecot: imap(dan): Panic: file imap-fetch.c: line 556 (imap_fetch_more): assertion failed: (ctx->client->output_cmd_lock == NULL || ctx->client->output_cmd_lock == cmd) Sep 6 03:39:32 mailjail dovecot: imap(dan): Fatal: master: service(imap): child 71153 killed with signal 6 (core not dumped - set service imap { drop_priv_before_exec=yes }) Sep 6 03:59:41 mailjail dovecot: imap(dan): Fatal: pool_system_realloc(2097152): Out of memory Sep 6 03:59:41 mailjail dovecot: imap(dan): Fatal: master: service(imap): child 67732 returned error 83 (Out of memory (service imap { vsz_limit=256 MB }, you may need to increase it) - set CORE_OUTOFMEM=1 environment to get core dump)
Check the message again - it says vsz_limit=256MB
Background: I’m in the only user on this system, but this server is accessed by my phone, my laptop, my tablet, and perhaps a web interface.
But there are some interesting values when I look at the output of doveconf. Specifically, vsz_limit is 18,447 PB… yeah, that’s pretty big.
service imap-login { vsz_limit = 18446744073709551615 B }
According to your log it was process "imap" what ran out of memory. You are showing configuration for imap-login - they are not the same thing, see http://wiki2.dovecot.org/Design/Processes
Not sure if this behaviour is a bug or something that is to be expected when you run out of VSZ limit in imap process - that needs to be answered by someone more knowledgeable
On Sep 7, 2014, at 4:39 AM, Jiri Bourek <bourek@thinline.cz> wrote:
On 6.9.2014 22:38, Dan Langille wrote:
I encountered this error on FreeBSD 9.3 with dovecot2-2.2.13_3
Short version; vsz_limit is 18,447 PB and it ran out of RAM. Suggestions for change?
Sep 6 03:39:32 mailjail dovecot: imap(dan): Panic: file imap-fetch.c: line 556 (imap_fetch_more): assertion failed: (ctx->client->output_cmd_lock == NULL || ctx->client->output_cmd_lock == cmd) Sep 6 03:39:32 mailjail dovecot: imap(dan): Fatal: master: service(imap): child 71153 killed with signal 6 (core not dumped - set service imap { drop_priv_before_exec=yes }) Sep 6 03:59:41 mailjail dovecot: imap(dan): Fatal: pool_system_realloc(2097152): Out of memory Sep 6 03:59:41 mailjail dovecot: imap(dan): Fatal: master: service(imap): child 67732 returned error 83 (Out of memory (service imap { vsz_limit=256 MB }, you may need to increase it) - set CORE_OUTOFMEM=1 environment to get core dump)
Check the message again - it says vsz_limit=256MB
Yes, I saw that…. I can’t explain why I didn’t question the rest of the output.
Background: I’m in the only user on this system, but this server is accessed by my phone, my laptop, my tablet, and perhaps a web interface.
But there are some interesting values when I look at the output of doveconf. Specifically, vsz_limit is 18,447 PB… yeah, that’s pretty big.
service imap-login { vsz_limit = 18446744073709551615 B }
According to your log it was process “im ap" what ran out of memory. You are showing configuration for imap-login - they are not the same thing, see http://wiki2.dovecot.org/Design/Processes
My errror, sorry. OK, here’s imap:
service imap { chroot = client_limit = 1 drop_priv_before_exec = no executable = imap extra_groups = group = idle_kill = 0 privileged_group = process_limit = 1024 process_min_avail = 0 protocol = imap service_count = 1 type = unix_listener login/imap { group = mode = 0666 user = } user = vsz_limit = 18446744073709551615 B }
Still the same value… why not 265MB?
Not sure if this behaviour is a bug or something that is to be expected when you run out of VSZ limit in imap process - that needs to be answered by someone more knowledgeable
It seems quite common:
$ doveconf | grep vsz_limit default_vsz_limit = 256 M vsz_limit = 18446744073709551615 B vsz_limit = 18446744073709551615 B vsz_limit = 18446744073709551615 B vsz_limit = 18446744073709551615 B vsz_limit = 18446744073709551615 B vsz_limit = 18446744073709551615 B vsz_limit = 18446744073709551615 B vsz_limit = 18446744073709551615 B vsz_limit = 18446744073709551615 B vsz_limit = 18446744073709551615 B vsz_limit = 18446744073709551615 B vsz_limit = 18446744073709551615 B vsz_limit = 18446744073709551615 B vsz_limit = 18446744073709551615 B vsz_limit = 18446744073709551615 B vsz_limit = 18446744073709551615 B vsz_limit = 18446744073709551615 B vsz_limit = 18446744073709551615 B vsz_limit = 18446744073709551615 B vsz_limit = 18446744073709551615 B vsz_limit = 18446744073709551615 B vsz_limit = 18446744073709551615 B vsz_limit = 18446744073709551615 B vsz_limit = 18446744073709551615 B [dan@mailjail:~] $
Thank you.
Background: I’m in the only user on this system, but this server is accessed by my phone, my laptop, my tablet, and perhaps a web interface.
But there are some interesting values when I look at the output of doveconf. Specifically, vsz_limit is 18,447 PB… yeah, that’s pretty big.
service imap-login { vsz_limit = 18446744073709551615 B }
According to your log it was process “im ap" what ran out of memory. You are showing configuration for imap-login - they are not the same thing, see http://wiki2.dovecot.org/Design/Processes
My errror, sorry. OK, here’s imap:
service imap { chroot = client_limit = 1 drop_priv_before_exec = no executable = imap extra_groups = group = idle_kill = 0 privileged_group = process_limit = 1024 process_min_avail = 0 protocol = imap service_count = 1 type = unix_listener login/imap { group = mode = 0666 user = } user = vsz_limit = 18446744073709551615 B }
Still the same value… why not 265MB?
Hm, I can only guess (someone with more knowledge of dovecot source code probably knows better), but it seems like that value is magic number for "not set". My doveconf outputs the same thing on services which don't have vsz_limit set explicitly, but when I list /proc/[pid]/limits on such a service, the row called "Max address space" (which is - AFAIK - limit on vsz) shows 2GB, a value of default_vsz_limit setting.
I'd say don't overthink it, just put explicit vsz_limit = 512M into service imap section (if your machine's RAM allows) and be done with it.
On 9/8/14 3:38 PM, Jiri Bourek wrote:> Hm, I can only guess (someone with more knowledge of dovecot source code
probably knows better), but it seems like that value is magic number for "not set". My doveconf outputs the same thing on services which don't have vsz_limit set explicitly, but when I list /proc/[pid]/limits on such a service, the row called "Max address space" (which is - AFAIK - limit on vsz) shows 2GB, a value of default_vsz_limit setting.
I'd say don't overthink it, just put explicit vsz_limit = 512M into service imap section (if your machine's RAM allows) and be done with it.
Right. According to my notes from when I upgraded to 2.x, "18446744073709551615" means it's not set and that the default_vsz_limit will apply, which is 256 MB.
To avoid the problem the original poster had, I used:
default_vsz_limit = 1 G
-- Robert L Mathews, Tiger Technologies, http://www.tigertech.net/
On Mon, Sep 08, 2014 at 04:41:18PM -0700, Robert L Mathews wrote:
On 9/8/14 3:38 PM, Jiri Bourek wrote:> Hm, I can only guess (someone with more knowledge of dovecot source code
probably knows better), but it seems like that value is magic number for "not set". My doveconf outputs the same thing on services which don't have vsz_limit set explicitly, but when I list /proc/[pid]/limits on such a service, the row called "Max address space" (which is - AFAIK - limit on vsz) shows 2GB, a value of default_vsz_limit setting.
I'd say don't overthink it, just put explicit vsz_limit = 512M into service imap section (if your machine's RAM allows) and be done with it.
Right. According to my notes from when I upgraded to 2.x, "18446744073709551615" means it's not set and that the default_vsz_limit will apply, which is 256 MB.
That seems sensible to me. 2^64 bytes is 18446744073709551616 (18.45EB), so the number shown is (2^64 - 1). As a *limit* for a 64-bit program, that seems about as close to "no limit" as you're going to get :)
To avoid the problem the original poster had, I used:
default_vsz_limit = 1 G
-- Robert L Mathews, Tiger Technologies, http://www.tigertech.net/
On Sep 8, 2014, at 6:38 PM, Jiri Bourek <bourek@thinline.cz> wrote:
Background: I’m in the only user on this system, but this server is accessed by my phone, my laptop, my tablet, and perhaps a web interface.
But there are some interesting values when I look at the output of doveconf. Specifically, vsz_limit is 18,447 PB… yeah, that’s pretty big.
service imap-login { vsz_limit = 18446744073709551615 B }
According to your log it was process “im ap" what ran out of memory. You are showing configuration for imap-login - they are not the same thing, see http://wiki2.dovecot.org/Design/Processes
My errror, sorry. OK, here’s imap:
service imap { chroot = client_limit = 1 drop_priv_before_exec = no executable = imap extra_groups = group = idle_kill = 0 privileged_group = process_limit = 1024 process_min_avail = 0 protocol = imap service_count = 1 type = unix_listener login/imap { group = mode = 0666 user = } user = vsz_limit = 18446744073709551615 B }
Still the same value… why not 265MB?
Hm, I can only guess (someone with more knowledge of dovecot source code probably knows better), but it seems like that value is magic number for "not set". My doveconf outputs the same thing on services which don't have vsz_limit set explicitly, but when I list /proc/[pid]/limits on such a service, the row called "Max address space" (which is - AFAIK - limit on vsz) shows 2GB, a value of default_vsz_limit setting.
I'd say don't overthink it, just put explicit vsz_limit = 512M into service imap section (if your machine's RAM allows) and be done with it.
For the record, the error has recurred a few times, so after today’s incident, I added this to dovecot.conf, and restarted dovecot:
service imap { vsz_limit = 512MB }
FWIW, the server contains 8192 MB of RAM.
On Sep 15, 2014, at 10:20 AM, Dan Langille <dan@langille.org> wrote:
On Sep 8, 2014, at 6:38 PM, Jiri Bourek <bourek@thinline.cz> wrote:
Background: I’m in the only user on this system, but this server is accessed by my phone, my laptop, my tablet, and perhaps a web interface.
But there are some interesting values when I look at the output of doveconf. Specifically, vsz_limit is 18,447 PB… yeah, that’s pretty big.
service imap-login { vsz_limit = 18446744073709551615 B }
According to your log it was process “im ap" what ran out of memory. You are showing configuration for imap-login - they are not the same thing, see http://wiki2.dovecot.org/Design/Processes
My errror, sorry. OK, here’s imap:
service imap { chroot = client_limit = 1 drop_priv_before_exec = no executable = imap extra_groups = group = idle_kill = 0 privileged_group = process_limit = 1024 process_min_avail = 0 protocol = imap service_count = 1 type = unix_listener login/imap { group = mode = 0666 user = } user = vsz_limit = 18446744073709551615 B }
Still the same value… why not 265MB?
Hm, I can only guess (someone with more knowledge of dovecot source code probably knows better), but it seems like that value is magic number for "not set". My doveconf outputs the same thing on services which don't have vsz_limit set explicitly, but when I list /proc/[pid]/limits on such a service, the row called "Max address space" (which is - AFAIK - limit on vsz) shows 2GB, a value of default_vsz_limit setting.
I'd say don't overthink it, just put explicit vsz_limit = 512M into service imap section (if your machine's RAM allows) and be done with it.
For the record, the error has recurred a few times, so after today’s incident, I added this to dovecot.conf, and restarted dovecot:
service imap { vsz_limit = 512MB }
FWIW, the server contains 8192 MB of RAM.
For the record, the problem has not recurred.
— Dan Langille
participants (4)
-
Dan Langille
-
Darac Marjal
-
Jiri Bourek
-
Robert L Mathews