[Dovecot] Panic: file mempool-system.c: line 104
Hello!
I have used Dovecot for more than a year without problems, but today it just crashed with this message:
dovecot: Apr 09 21:23:38 Panic: file mempool-system.c: line 104 (pool_system_realloc): assertion failed: (old_size == (size_t)-1 || mem == NULL || old_size <= malloc_usable_size(mem)) dovecot: Apr 09 21:23:38 Error: Raw backtrace: /usr/local/sbin/dovecot [0x805757c] -> /usr/local/sbin/dovecot [0x8057664] -> /usr/local/sbin/dovecot [0x80514e9] -> /usr/local/sbin/dovecot [0x8056edb] -> /usr/local/sbin/dovecot [0x805bcc1] -> /usr/local/sbin/dovecot [0x8055bbf] -> /usr/local/sbin/dovecot [0x8055d3b] -> /usr/local/sbin/dovecot [0x8055d87] -> /usr/local/sbin/dovecot [0x8062448] -> /usr/local/sbin/dovecot [0x805aadf] -> /usr/local/sbin/dovecot [0x805a5a7] -> /usr/local/sbin/dovecot [0x804d63a] -> /usr/local/sbin/dovecot [0x804da67] -> /usr/local/sbin/dovecot [0x804e4ce] -> /usr/local/sbin/dovecot [0x805a17a] -> /usr/local/sbin/dovecot [0x805adac] -> /usr/local/sbin/dovecot [0x8059f89] -> /usr/local/sbin/dovecot [0x8051b43] -> /lib/libc.so.6(__libc_start_main+0xd8) [0xb7e49df8] -> /usr/local/sbin/dovecot [0x804a451]
dovecot --version 1.1.7
It seems, than some brute force attack caused this, here some more log:
dovecot: Apr 09 21:23:36 Info: auth(default): client out: FAIL 1 user=jennie dovecot: Apr 09 21:23:36 Info: auth(default): client out: FAIL 1 user=jeffrey dovecot: Apr 09 21:23:36 Info: auth(default): client out: FAIL 1 user=job dovecot: Apr 09 21:23:36 Info: auth-worker(default): sql(jimmy,81.149.214.201): query: SELECT password FROM mailbox WHERE username = 'jimmy' AND active = '1' dovecot: Apr 09 21:23:36 Info: auth(default): client in: AUTH 1 PLAIN service=pop3 lip=195.3.144.29 rip=81.149.214.201 lport=110 rport=34646 resp=<hidden> dovecot: Apr 09 21:23:36 Info: auth-worker(default): sql(jimmy,81.149.214.201): unknown user
For now I blocked this one IP (81.149.214.201), but wondering if this could happen again in the future?
Dont really understand whats the problem. I have ~2Gb of memory free,
so it can not be a out of memory issue.
Any ideas?
Thank you!
-- Martins Lazdans
On Thu, 2009-04-09 at 21:37 +0300, Martins Lazdans wrote:
Hello!
I have used Dovecot for more than a year without problems, but today it just crashed with this message:
dovecot: Apr 09 21:23:38 Panic: file mempool-system.c: line 104 (pool_system_realloc): assertion failed: (old_size == (size_t)-1 || mem == NULL || old_size <= malloc_usable_size(mem))
Did Dovecot happen to write a core file somewhere? Should be in base_dir (e.g. /var/run/dovecot)? A gdb backtrace would be helpful.
No, there is no core dumps. I believe I can somehow enable to dump core?
Timo Sirainen wrote:
On Thu, 2009-04-09 at 21:37 +0300, Martins Lazdans wrote:
Hello!
I have used Dovecot for more than a year without problems, but today it just crashed with this message:
dovecot: Apr 09 21:23:38 Panic: file mempool-system.c: line 104 (pool_system_realloc): assertion failed: (old_size == (size_t)-1 || mem == NULL || old_size <= malloc_usable_size(mem))
Did Dovecot happen to write a core file somewhere? Should be in base_dir (e.g. /var/run/dovecot)? A gdb backtrace would be helpful.
-- Martins Lazdans
Run ulimit -c unlimited before starting dovecot. And note that several distributions override the ulimit if you use /etc/init.d/dovecot to start it. Newer Dovecot versions (v1.1.9+) say at startup if core dumps are disabled.
On Thu, 2009-04-09 at 23:06 +0300, Martins Lazdans wrote:
No, there is no core dumps. I believe I can somehow enable to dump core?
Timo Sirainen wrote:
On Thu, 2009-04-09 at 21:37 +0300, Martins Lazdans wrote:
Hello!
I have used Dovecot for more than a year without problems, but today it just crashed with this message:
dovecot: Apr 09 21:23:38 Panic: file mempool-system.c: line 104 (pool_system_realloc): assertion failed: (old_size == (size_t)-1 || mem == NULL || old_size <= malloc_usable_size(mem))
Did Dovecot happen to write a core file somewhere? Should be in base_dir (e.g. /var/run/dovecot)? A gdb backtrace would be helpful.
Ok, I unblocked that IP and started Dovecot with
ulimit -c unlimited && /usr/local/sbin/dovecot
and will see if that IP will return.
But I totally forgot I am using Google`s perf-tools (http://code.google.com/p/google-perftools/) tcmalloc library. Maybe that library is faulty and/or incompatible with Dovecot?
export LD_PRELOAD=/usr/local/lib/libtcmalloc.so
However, I switched to Google`s libtcmalloc.so long time ago, and dovecot was running just fine until today. MySQL, nginx, clamav and postfix are running fine too.
P.S. I got 2 responses from you :)
Timo Sirainen wrote:
Run ulimit -c unlimited before starting dovecot. And note that several distributions override the ulimit if you use /etc/init.d/dovecot to start it. Newer Dovecot versions (v1.1.9+) say at startup if core dumps are disabled.
On Thu, 2009-04-09 at 23:06 +0300, Martins Lazdans wrote:
No, there is no core dumps. I believe I can somehow enable to dump core?
Timo Sirainen wrote:
On Thu, 2009-04-09 at 21:37 +0300, Martins Lazdans wrote:
Hello!
I have used Dovecot for more than a year without problems, but today it just crashed with this message:
dovecot: Apr 09 21:23:38 Panic: file mempool-system.c: line 104 (pool_system_realloc): assertion failed: (old_size == (size_t)-1 || mem == NULL || old_size <= malloc_usable_size(mem)) Did Dovecot happen to write a core file somewhere? Should be in base_dir (e.g. /var/run/dovecot)? A gdb backtrace would be helpful.
-- Martins Lazdans
On Thu, 2009-04-09 at 23:41 +0300, Martins Lazdans wrote:
Ok, I unblocked that IP and started Dovecot with
ulimit -c unlimited && /usr/local/sbin/dovecotand will see if that IP will return.But I totally forgot I am using Google`s perf-tools (http://code.google.com/p/google-perftools/) tcmalloc library. Maybe that library is faulty and/or incompatible with Dovecot?
Oh. That's the most likely reason. Dovecot is using this non-standard malloc_usable_size() function. Maybe your tcmalloc doesn't implement it and instead glibc's version gets called which then causes it to return wrong size. And the reason why this happens rarely is because Dovecot rarely calls realloc() function.
export LD_PRELOAD=/usr/local/lib/libtcmalloc.so
Dovecot clears environment when calling child processes, so I think this only makes it use the tcmalloc in dovecot master process.
Anyway if you want to keep using tcmalloc, you could comment out HAVE_MALLOC_USABLE_SIZE from config.h and reinstall.
participants (2)
-
Martins Lazdans
-
Timo Sirainen