[Dovecot] Out of memory [repost as a new thread]
Hi,
this happens since a few days on a Gentoo hardened system using a grsecurity enabled kernel running Dovecot 1.0.10, only to 2 of 10 users though:
--8<--
kernel: grsec: From 192.168.0.1: denied resource overstep by
requesting 537325568 for RLIMIT_AS against limit 536870912
for /usr/libexec/dovecot/imap[imap:15708] uid/euid:30010/30010
gid/egid:30006/30006, parent /usr/sbin/dovecot[dovecot:15574]
uid/euid:0/0 gid/egid:0/0
kernel: grsec: From 192.168.0.1: denied resource overstep by
requesting 537321472 for RLIMIT_AS against limit 536870912
for /usr/libexec/dovecot/imap[imap:15708] uid/euid:30010/30010
gid/egid:30006/30006, parent /usr/sbin/dovecot[dovecot:15574]
uid/euid:0/0 gid/egid:0/0
kernel: grsec: From 192.168.0.1: denied resource overstep by
requesting 537456640 for RLIMIT_AS against limit 536870912
for /usr/libexec/dovecot/imap[imap:15708] uid/euid:30010/30010
gid/egid:30006/30006, parent /usr/sbin/dovecot[dovecot:15574]
uid/euid:0/0 gid/egid:0/0
kernel: grsec: From 192.168.0.1: denied resource overstep by
requesting 537321472 for RLIMIT_AS against limit 536870912
for /usr/libexec/dovecot/imap[imap:15708] uid/euid:30010/30010
gid/egid:30006/30006, parent /usr/sbin/dovecot[dovecot:15574]
uid/euid:0/0 gid/egid:0/0
dovecot: IMAP(info): block_alloc(): Out of memory
dovecot: child 15708 (imap) returned error 83 (Out of memory)
--8<--
grsecurity only logs the attempted resource overstep.
I already increased mail_process_size to 512M and deleted the index.cache* files inside the users maildir -- didn't help :/
Any idea why this is happening or how I could find out?
TIA :)
Regards, Wolfram Schlich wschlich@gentoo.org Gentoo Linux * http://dev.gentoo.org/~wschlich/
On Tue, 2008-01-15 at 14:51 +0100, Wolfram Schlich wrote:
this happens since a few days on a Gentoo hardened system using a grsecurity enabled kernel running Dovecot 1.0.10, only to 2 of 10 users though:
--8<--
kernel: grsec: From 192.168.0.1: denied resource overstep by
requesting 537325568 for RLIMIT_AS against limit 536870912 \
If it's trying to allocate 500MB of memory, there's a bug somewhere. Could you get gdb backtrace of this? The attached patch changes Dovecot to call abort() so it should write core files to user's home dir. See http://dovecot.org/bugreport.html
- Timo Sirainen tss@iki.fi [2008-01-20 08:49]:
On Tue, 2008-01-15 at 14:51 +0100, Wolfram Schlich wrote:
this happens since a few days on a Gentoo hardened system using a grsecurity enabled kernel running Dovecot 1.0.10, only to 2 of 10 users though:
--8<--
kernel: grsec: From 192.168.0.1: denied resource overstep by
requesting 537325568 for RLIMIT_AS against limit 536870912 \If it's trying to allocate 500MB of memory, there's a bug somewhere. Could you get gdb backtrace of this? The attached patch changes Dovecot to call abort() so it should write core files to user's home dir. See http://dovecot.org/bugreport.html
Hmm. I tried your patch and compiled dovecot with --enable-debug, enabled coredumps for the service (with ulimit -c unlimited) and I got some corefiles now.
/usr/libexec/dovecot/imap: ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), for GNU/Linux 2.6.9, not stripped
--8<-- uluru tmp # gdb /usr/libexec/dovecot/imap core.x\{imap\}.u\{30010\}.g\{30006\}.p\{20880\}.t\{1201094907\} GNU gdb 6.7.1 Copyright (C) 2007 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "i686-pc-linux-gnu"... (no debugging symbols found) Using host libthread_db library "/lib/libthread_db.so.1". (no debugging symbols found) (no debugging symbols found) Core was generated by `imap [info ::ffff:172.16.223.18]'. Program terminated with signal 11, Segmentation fault. #0 0x169917d5 in ?? () from /lib/ld-linux.so.2 (gdb) bt full #0 0x169917d5 in ?? () from /lib/ld-linux.so.2 No symbol table info available. #1 0x169bfba9 in ?? () No symbol table info available. #2 0xb146d6ff in ?? () No symbol table info available. #3 0x00000001 in ?? () No symbol table info available. #4 0x00000000 in ?? () No symbol table info available. (gdb) quit uluru tmp # --8<--
Looks like this happens in glibc, right?! I haven't compiled glibc with debugging symbols/unstripped so far of course...
Regards, Wolfram Schlich wschlich@gentoo.org Gentoo Linux * http://dev.gentoo.org/~wschlich/
On Wed, 2008-01-23 at 15:29 +0100, Wolfram Schlich wrote:
I tried your patch and compiled dovecot with --enable-debug,
There's no need for --enable-debug. It's mainly useful for developers when developing new code.
/usr/libexec/dovecot/imap: ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), for GNU/Linux 2.6.9, not stripped
--8<-- uluru tmp # gdb /usr/libexec/dovecot/imap core.x\{imap\}.u\{30010\}.g\{30006\}.p\{20880\}.t\{1201094907\}
Looks ok..
(gdb) bt full #0 0x169917d5 in ?? () from /lib/ld-linux.so.2 No symbol table info available. #1 0x169bfba9 in ?? () No symbol table info available. #2 0xb146d6ff in ?? () No symbol table info available. #3 0x00000001 in ?? () No symbol table info available. #4 0x00000000 in ?? () No symbol table info available.
Unfortunately this is broken. Even those addresses are impossible (#4 shows that one call was from NULL pointer, #3 is 1). This just seems to happen sometimes for core dumps, possibly even repeatedly..
The one sure way to get a usable backtrace would be to attach gdb to a running imap process, but that would require you to be able to reproduce the bug at a specific time (so you can attach gdb, then make it crash)..
Or maybe if you have more core dumps, one of them shows something? The important thing is that if it only shows "??" it's broken. At the very least the last lines should contain something like:
#2 0x00000000004881cd in io_loop_run (ioloop=0x6c9d30) at ioloop.c:301 #3 0x0000000000424a62 in main (argc=<value optimized out>, argv=0x7fff1841b9b8, envp=0x7fff1841b9c8) at main.c:293
- Timo Sirainen tss@iki.fi [2008-01-23 16:46]:
On Wed, 2008-01-23 at 15:29 +0100, Wolfram Schlich wrote:
I tried your patch and compiled dovecot with --enable-debug,
There's no need for --enable-debug. It's mainly useful for developers when developing new code.
Ok.
/usr/libexec/dovecot/imap: ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), for GNU/Linux 2.6.9, not stripped
--8<-- uluru tmp # gdb /usr/libexec/dovecot/imap core.x\{imap\}.u\{30010\}.g\{30006\}.p\{20880\}.t\{1201094907\}
Looks ok..
(gdb) bt full #0 0x169917d5 in ?? () from /lib/ld-linux.so.2 No symbol table info available. #1 0x169bfba9 in ?? () No symbol table info available. #2 0xb146d6ff in ?? () No symbol table info available. #3 0x00000001 in ?? () No symbol table info available. #4 0x00000000 in ?? () No symbol table info available.
Unfortunately this is broken. Even those addresses are impossible (#4 shows that one call was from NULL pointer, #3 is 1). This just seems to happen sometimes for core dumps, possibly even repeatedly..
The one sure way to get a usable backtrace would be to attach gdb to a running imap process, but that would require you to be able to reproduce the bug at a specific time (so you can attach gdb, then make it crash)..
Or maybe if you have more core dumps, one of them shows something? The important thing is that if it only shows "??" it's broken. At the very least the last lines should contain something like:
#2 0x00000000004881cd in io_loop_run (ioloop=0x6c9d30) at ioloop.c:301 #3 0x0000000000424a62 in main (argc=<value optimized out>, argv=0x7fff1841b9b8, envp=0x7fff1841b9c8) at main.c:293
No chance so far. I even recompiled glibc and kept the debug symbols: --8<-- GNU gdb 6.7.1 Copyright (C) 2007 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "i686-pc-linux-gnu"... (no debugging symbols found) Using host libthread_db library "/lib/libthread_db.so.1". (no debugging symbols found) (no debugging symbols found) Core was generated by `imap [someuser ::ffff:192.168.1.1]'. Program terminated with signal 11, Segmentation fault. #0 0x1a2887d5 in _start () from /lib/ld-linux.so.2 (gdb) bt full #0 0x1a2887d5 in _start () from /lib/ld-linux.so.2 No symbol table info available. #1 0x1a2a88f1 in ?? () No symbol table info available. #2 0xb136ec5f in ?? () No symbol table info available. #3 0x00000001 in ?? () No symbol table info available. #4 0x00000000 in ?? () No symbol table info available. (gdb) --8<--
I have to say this machine is a Gentoo Hardened machine using a PaX kernel and PIE/SSP userland...
Also I noticed that it doesn't *always* core: --8<-- 2008-01-24 05:16:00 +01:00; uluru; kern.alert; kernel: grsec: From 192.168.1.1: denied resource overstep by requesting 1079001088 for RLIMIT_AS against limit 1073741824 for /usr/libexec/dovecot/imap[imap:22235] uid/euid:30001/30001 gid/egid:30006/30006, parent /usr/sbin/dovecot[dovecot:20924] uid/euid:0/0 gid/egid:0/0 2008-01-24 05:16:00 +01:00; uluru; kern.alert; kernel: grsec: From 192.168.1.1: signal 11 sent to /usr/libexec/dovecot/imap[imap:22235] uid/euid:30001/30001 gid/egid:30006/30006, parent /usr/sbin/dovecot[dovecot:20924] uid/euid:0/0 gid/egid:0/0 2008-01-24 05:16:00 +01:00; uluru; mail.err; dovecot: child 22235 (imap) killed with signal 11
2008-01-24 11:17:42 +01:00; uluru; mail.err; dovecot: IMAP(info): pool_system_malloc(): Out of memory 2008-01-24 11:17:42 +01:00; uluru; mail.err; dovecot: child 1964 (imap) returned error 83 (Out of memory) 2008-01-24 11:17:42 +01:00; uluru; kern.alert; kernel: grsec: From 192.168.1.1: denied resource overstep by requesting 1073864704 for RLIMIT_AS against limit 1073741824 for /usr/libexec/dovecot/imap[imap:1964] uid/euid:30010/30010 gid/egid:30006/30006, parent /usr/sbin/dovecot[dovecot:20924] uid/euid:0/0 gid/egid:0/0
2008-01-24 10:58:42 +01:00; uluru; kern.alert; kernel: grsec: From 192.168.1.1: denied resource overstep by requesting 1074552832 for RLIMIT_AS against limit 1073741824 for /usr/libexec/dovecot/imap[imap:28392] uid/euid:30001/30001 gid/egid:30006/30006, parent /usr/sbin/dovecot[dovecot:20924] uid/euid:0/0 gid/egid:0/0 2008-01-24 10:58:42 +01:00; uluru; mail.err; dovecot: IMAP(someuser): pool_system_malloc(): Out of memory 2008-01-24 10:58:42 +01:00; uluru; mail.err; dovecot: child 28392 (imap) returned error 83 (Out of memory) --8<--
So, only the crash from 05:16 produced a core, the others didn't... WTF?! :)) And yes, I restarted dovecot right after I installed the patched version...
Regards, Wolfram Schlich wschlich@gentoo.org Gentoo Linux * http://dev.gentoo.org/~wschlich/
On Thu, 2008-01-24 at 11:47 +0100, Wolfram Schlich wrote:
No symbol table info available. #1 0x1a2a88f1 in ?? () No symbol table info available. #2 0xb136ec5f in ?? () No symbol table info available. #3 0x00000001 in ?? () No symbol table info available. #4 0x00000000 in ?? () No symbol table info available. (gdb) --8<--
I have to say this machine is a Gentoo Hardened machine using a PaX kernel and PIE/SSP userland...
I remember I stopped using PaX a few years ago because I had trouble with gdb, but I don't remember if it was the same problem.
2008-01-24 05:16:00 +01:00; uluru; mail.err; dovecot: child 22235 (imap) killed with signal 11 2008-01-24 11:17:42 +01:00; uluru; mail.err; dovecot: IMAP(info): pool_system_malloc(): Out of memory 2008-01-24 10:58:42 +01:00; uluru; mail.err; dovecot: IMAP(someuser): pool_system_malloc(): Out of memory
So, only the crash from 05:16 produced a core, the others didn't... WTF?! :)) And yes, I restarted dovecot right after I installed the patched version...
Interesting. My patch was only for pool_alloconly_malloc(), because I thought the problem was with a single memory allocation call. But these two show that it also happens in another place. Maybe these really have some email (or something) that causes Dovecot to eat lots of memory?
Also the signal 11 is a different problem from these memory allocation problems.
But unfortunately without a usable backtrace or a way to reproduce this, I can't really do anything..
- Timo Sirainen tss@iki.fi [2008-01-24 13:50]:
On Thu, 2008-01-24 at 11:47 +0100, Wolfram Schlich wrote:
I have to say this machine is a Gentoo Hardened machine using a PaX kernel and PIE/SSP userland...
I remember I stopped using PaX a few years ago because I had trouble with gdb, but I don't remember if it was the same problem. [...] But unfortunately without a usable backtrace or a way to reproduce this, I can't really do anything..
Ok, I got something now after consulting these pages: http://www.gentoo.org/proj/en/qa/backtraces.xml http://www.gentoo.org/proj/en/hardened/hardenedfaq.xml#hardeneddebug Maybe you can link to those from the wiki, might be worth mentioning for other Gentoo Hardened users as well :)
So, I have a 115 lines long backtrace which also contains some non-printable characters. Should I send you the file in a private mail?
Thanks, Wolfram
participants (2)
-
Timo Sirainen
-
Wolfram Schlich