[Dovecot] SIGSEGV login process
Hi all. I recently upgraded from 1.0.0 to 1.0.7 version and found that process imap-login begins randomly crash:
pid 29997 (imap-login), uid 143: exited on signal 11 (core dumped) pid 32189 (imap-login), uid 143: exited on signal 11 (core dumped) pid 32121 (imap-login), uid 143: exited on signal 11 (core dumped) pid 31309 (imap-login), uid 143: exited on signal 11 (core dumped)
i guess that it happens during high load on CPU (but just guess) no anything upon upgrade changed - traffic, config are the same. can anyone help me? running FreeBSD 6.2, compiled from port with mysql, lda, kqueue, pop3 support.
mail# dovecot -n
1.0.7: /usr/local/etc/dovecot.conf
protocols: imap pop3 pop3s imaps listen(default): *:143 listen(imap): *:143 listen(pop3): *:110 ssl_listen(default): *:993 ssl_listen(imap): *:993 ssl_listen(pop3): *:995 ssl_cert_file: /usr/local/etc/ssl_mail/dovecot.pem ssl_key_file: /usr/local/etc/ssl_mail/dovecot.pem ssl_cipher_list: ALL:!LOW disable_plaintext_auth: no login_dir: /var/run/dovecot/login login_executable(default): /usr/local/libexec/dovecot/imap-login login_executable(imap): /usr/local/libexec/dovecot/imap-login login_executable(pop3): /usr/local/libexec/dovecot/pop3-login verbose_proctitle: yes first_valid_uid: 26 last_valid_uid: 26 mail_extra_groups: mail mail_location: maildir:~/Maildir mmap_disable: yes mail_executable(default): /usr/local/libexec/dovecot/imap mail_executable(imap): /usr/local/libexec/dovecot/imap mail_executable(pop3): /usr/local/libexec/dovecot/pop3 mail_plugins(default): quota imap_quota mail_plugins(imap): quota imap_quota mail_plugins(pop3): quota mail_plugin_dir(default): /usr/local/lib/dovecot/imap mail_plugin_dir(imap): /usr/local/lib/dovecot/imap mail_plugin_dir(pop3): /usr/local/lib/dovecot/pop3 imap_client_workarounds(default): delay-newmail outlook-idle netscape-eoh tb-extra-mailbox-sep imap_client_workarounds(imap): delay-newmail outlook-idle netscape-eoh tb-extra-mailbox-sep imap_client_workarounds(pop3): outlook-idle pop3_uidl_format(default): pop3_uidl_format(imap): pop3_uidl_format(pop3): %08Xu%08Xv pop3_client_workarounds(default): pop3_client_workarounds(imap): pop3_client_workarounds(pop3): outlook-no-nuls oe-ns-eoh auth default: mechanisms: plain cram-md5 digest-md5 apop login default_realm: proc.ru passdb: driver: sql args: /usr/local/etc/dovecot-sql.conf userdb: driver: sql args: /usr/local/etc/dovecot-sql.conf
mail# grep -v '^ *\(#.*\)\?$' /usr/local/etc/dovecot-sql.conf driver = mysql connect = host=/tmp/mysql.sock dbname=exim_users user=exim password=** default_pass_scheme = PLAIN password_query = SELECT concat(pw_name, '@', domain) as user, pw_clear_passwd as password FROM vusers WHERE pw_name = '%n' AND domain = '%d' user_query = SELECT pw_dir AS home, 26 AS uid, 26 AS gid, concat('maildir:storage=', quota*1024) AS quota FROM vusers WHERE pw_name = '%n' AND domain = '%d'
backtrace from core file:
mail# gdb ./work/dovecot-1.0.7/src/imap-login/imap-login /var/run/dovecot/login/tmp/imap-login.core GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd"...(no debugging symbols found)...
warning: exec file is newer than core file. Core was generated by `imap-login'. Program terminated with signal 11, Segmentation fault. Reading symbols from /usr/lib/libssl.so.4...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libssl.so.4 Reading symbols from /lib/libcrypto.so.4...(no debugging symbols found)...done. Loaded symbols for /lib/libcrypto.so.4 Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/libc.so.6 Reading symbols from /libexec/ld-elf.so.1...(no debugging symbols found)...done. Loaded symbols for /libexec/ld-elf.so.1 #0 0x08051e2d in auth_client_request_continue () (gdb) bt #0 0x08051e2d in auth_client_request_continue () #1 0x0804b8db in client_auth_input () #2 0x080562ec in io_loop_handler_run () #3 0x0805599d in io_loop_run () #4 0x0804d963 in main () (gdb)
or what anything info should i provide?
On Thu, 2007-11-22 at 21:16 +0300, Maxim wrote:
#0 0x08051e2d in auth_client_request_continue () (gdb) bt #0 0x08051e2d in auth_client_request_continue () #1 0x0804b8db in client_auth_input () #2 0x080562ec in io_loop_handler_run () #3 0x0805599d in io_loop_run () #4 0x0804d963 in main () (gdb)
or what anything info should i provide?
Any chance of getting the backtrace with debug symbols enabled? If you compile from sources, they're in the binaries by default.
Also it sometimes helps backtraces if optimizations are disabled. You can do this with:
CFLAGS=-g ./configure
On Mon, 26 Nov 2007 15:04:06 +0200, Timo Sirainen <tss@iki.fi> wrote:
On Thu, 2007-11-22 at 21:16 +0300, Maxim wrote:
#0 0x08051e2d in auth_client_request_continue () (gdb) bt #0 0x08051e2d in auth_client_request_continue () #1 0x0804b8db in client_auth_input () #2 0x080562ec in io_loop_handler_run () #3 0x0805599d in io_loop_run () #4 0x0804d963 in main () (gdb)
or what anything info should i provide?
Any chance of getting the backtrace with debug symbols enabled? If you compile from sources, they're in the binaries by default.
Also it sometimes helps backtraces if optimizations are disabled. You can do this with:
CFLAGS=-g ./configure
i found problem.
i have installed exim with self delivery (not dovecot LDA) and just install from port mail/dovecot-sieve and have these crashes. e.g. finally case is:
- install exim with Maildir delivery
- install dovecot
- start both of them
- install mail/dovecot-sieve
- not touch anything, just wait high load (LA~10..20)
- login processes begins randomly crashes
did you still need debug core+bt?
On Mon, 2007-11-26 at 16:13 +0300, Maxim Lougovsky wrote:
i have installed exim with self delivery (not dovecot LDA) and just install from port mail/dovecot-sieve and have these crashes. e.g. finally case is:
- install exim with Maildir delivery
- install dovecot
- start both of them
- install mail/dovecot-sieve
- not touch anything, just wait high load (LA~10..20)
So the high load is the point I guess.
- login processes begins randomly crashes
did you still need debug core+bt?
It would help a lot. Otherwise it'll probably take me quite a lot of time to reproduce it, if I'm able to do it at all (it could be that some clients do something special that causes it).
On Mon, 26 Nov 2007 15:22:29 +0200, Timo Sirainen <tss@iki.fi> wrote:
On Mon, 2007-11-26 at 16:13 +0300, Maxim Lougovsky wrote:
i have installed exim with self delivery (not dovecot LDA) and just install from port mail/dovecot-sieve and have these crashes. e.g. finally case is:
- install exim with Maildir delivery
- install dovecot
- start both of them
- install mail/dovecot-sieve
- not touch anything, just wait high load (LA~10..20)
So the high load is the point I guess.
- login processes begins randomly crashes
did you still need debug core+bt?
It would help a lot. Otherwise it'll probably take me quite a lot of time to reproduce it, if I'm able to do it at all (it could be that some clients do something special that causes it).
ok, i soon send to list backtrace. Thank you
On Mon, 26 Nov 2007 16:35:46 +0300, Maxim Lougovsky <h-k@proc.ru> wrote:
On Mon, 26 Nov 2007 15:22:29 +0200, Timo Sirainen <tss@iki.fi> wrote:
On Mon, 2007-11-26 at 16:13 +0300, Maxim Lougovsky wrote:
i have installed exim with self delivery (not dovecot LDA) and just install from port mail/dovecot-sieve and have these crashes. e.g. finally case is:
- install exim with Maildir delivery
- install dovecot
- start both of them
- install mail/dovecot-sieve
- not touch anything, just wait high load (LA~10..20)
So the high load is the point I guess.
- login processes begins randomly crashes
did you still need debug core+bt?
It would help a lot. Otherwise it'll probably take me quite a lot of time to reproduce it, if I'm able to do it at all (it could be that some clients do something special that causes it).
ok, i soon send to list backtrace. Thank you
Hmm, really strange. now bt shown other function rather previous:
mail# gdb ./work/dovecot-1.0.7/src/imap-login/imap-login /tmp/imap-login.core GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd"...
warning: exec file is newer than core file. Core was generated by `imap-login'. Program terminated with signal 11, Segmentation fault. #0 0x08051e2d in imap_parser_read_arg (parser=0x0) at imap-parser.c:518 518 data = i_stream_get_data(parser->input, &data_size); (gdb) bt #0 0x08051e2d in imap_parser_read_arg (parser=0x0) at imap-parser.c:518 #1 0x0804b8db in client_unref (client=0x806ea00) at client.c:536 #2 0x080562ec in fd_debug_verify_leaks (first_fd=134705600, last_fd=134598854) at fd-close-on-exec.c:56 #3 0x0805599d in i_error (format=0x80771c0 "") at failures.c:210 #4 0x0804d963 in login_proxy_free (proxy=0x1) at login-proxy.c:207 #5 0x0804a6a6 in _start () #6 0x00000001 in ?? () (gdb)
On Mon, 2007-11-26 at 16:51 +0300, Maxim Lougovsky wrote:
Hmm, really strange. now bt shown other function rather previous: .. mail# gdb ./work/dovecot-1.0.7/src/imap-login/imap-login /tmp/imap-login.core .. warning: exec file is newer than core file.
The core file needs to used against the exact same binary that produced it. Otherwise the backtrace is corrupted. So you'd have to install the imap-login and get a new core file..
On Mon, 26 Nov 2007 15:59:38 +0200, Timo Sirainen <tss@iki.fi> wrote:
On Mon, 2007-11-26 at 16:51 +0300, Maxim Lougovsky wrote:
Hmm, really strange. now bt shown other function rather previous: .. mail# gdb ./work/dovecot-1.0.7/src/imap-login/imap-login /tmp/imap-login.core .. warning: exec file is newer than core file.
The core file needs to used against the exact same binary that produced it. Otherwise the backtrace is corrupted. So you'd have to install the imap-login and get a new core file..
ok.
On Thu, 2007-11-22 at 21:16 +0300, Maxim wrote:
#0 0x08051e2d in auth_client_request_continue () (gdb) bt #0 0x08051e2d in auth_client_request_continue () #1 0x0804b8db in client_auth_input ()
Actually does this patch fix it? http://hg.dovecot.org/dovecot-1.0/raw-rev/73a3a6b1af36
On Wed, 28 Nov 2007 09:45:45 +0200, Timo Sirainen <tss@iki.fi> wrote:
On Thu, 2007-11-22 at 21:16 +0300, Maxim wrote:
#0 0x08051e2d in auth_client_request_continue () (gdb) bt #0 0x08051e2d in auth_client_request_continue () #1 0x0804b8db in client_auth_input ()
Actually does this patch fix it? http://hg.dovecot.org/dovecot-1.0/raw-rev/73a3a6b1af36
sorry, am still unable reproduce it. but trying :)
On Wed, 28 Nov 2007 10:50:01 +0300, Maxim Lougovsky <h-k@proc.ru> wrote:
On Wed, 28 Nov 2007 09:45:45 +0200, Timo Sirainen <tss@iki.fi> wrote:
On Thu, 2007-11-22 at 21:16 +0300, Maxim wrote:
#0 0x08051e2d in auth_client_request_continue () (gdb) bt #0 0x08051e2d in auth_client_request_continue () #1 0x0804b8db in client_auth_input ()
Actually does this patch fix it? http://hg.dovecot.org/dovecot-1.0/raw-rev/73a3a6b1af36
sorry, am still unable reproduce it. but trying :)
got it:
mail# gdb /usr/local/libexec/dovecot/imap-login /var/run/dovecot/login/tmp/imap-login.core GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd"... Core was generated by `imap-login'. Program terminated with signal 11, Segmentation fault. Reading symbols from /usr/lib/libssl.so.4...done. Loaded symbols for /usr/lib/libssl.so.4 Reading symbols from /lib/libcrypto.so.4...done. Loaded symbols for /lib/libcrypto.so.4 Reading symbols from /lib/libc.so.6...done. Loaded symbols for /lib/libc.so.6 Reading symbols from /libexec/ld-elf.so.1...done. Loaded symbols for /libexec/ld-elf.so.1 #0 0x08054096 in auth_client_request_continue (request=0x0, data_base64=0x808806b "ns1 NAMESPACE") at auth-server-request.c:377 377 auth-server-request.c: No such file or directory. in auth-server-request.c (gdb) bt #0 0x08054096 in auth_client_request_continue (request=0x0, data_base64=0x808806b "ns1 NAMESPACE") at auth-server-request.c:377 #1 0x0804beef in client_auth_input (context=0x8074a00) at client-authenticate.c:72 #2 0x08059afe in io_loop_handler_run (ioloop=0x807d1c0) at ioloop-kqueue.c:153 #3 0x08058f0e in io_loop_run (ioloop=0x807d1c0) at ioloop.c:329 #4 0x0804e905 in main (argc=1, argv=0xbfbfec2c, envp=0xbfbfec34) at main.c:432
On Fri, 2007-11-30 at 10:19 +0300, Maxim Lougovsky wrote:
got it: .. #0 0x08054096 in auth_client_request_continue (request=0x0, data_base64=0x808806b "ns1 NAMESPACE") at auth-server-request.c:377 377 auth-server-request.c: No such file or directory. in auth-server-request.c
So the client sent "ns1 NAMESPACE" command before Dovecot had finished handling login. I guess this crash was with 1.0.7? I think there's a very good chance that 1.0.8 has fixed this.
On Fri, 30 Nov 2007 14:47:24 +0200, Timo Sirainen <tss@iki.fi> wrote:
On Fri, 2007-11-30 at 10:19 +0300, Maxim Lougovsky wrote:
got it: .. #0 0x08054096 in auth_client_request_continue (request=0x0, data_base64=0x808806b "ns1 NAMESPACE") at auth-server-request.c:377 377 auth-server-request.c: No such file or directory. in auth-server-request.c
So the client sent "ns1 NAMESPACE" command before Dovecot had finished handling login. I guess this crash was with 1.0.7? I think there's a very good chance that 1.0.8 has fixed this.
10X. i will try it and report about results.
On Fri, 30 Nov 2007 16:01:23 +0300, Maxim Lougovsky <h-k@proc.ru> wrote:
On Fri, 30 Nov 2007 14:47:24 +0200, Timo Sirainen <tss@iki.fi> wrote:
On Fri, 2007-11-30 at 10:19 +0300, Maxim Lougovsky wrote:
got it: .. #0 0x08054096 in auth_client_request_continue (request=0x0, data_base64=0x808806b "ns1 NAMESPACE") at auth-server-request.c:377 377 auth-server-request.c: No such file or directory. in auth-server-request.c
So the client sent "ns1 NAMESPACE" command before Dovecot had finished handling login. I guess this crash was with 1.0.7? I think there's a very good chance that 1.0.8 has fixed this.
10X. i will try it and report about results.
Seems to be problem with 1.0.8 build solved. Thank you.
participants (3)
-
Maxim
-
Maxim Lougovsky
-
Timo Sirainen