On Feb 19, 2010, at 9:23 PM, Timo Sirainen wrote:

On Fri, 2010-02-19 at 15:28 -0600, Todd Rinaldo wrote:

...

pop3-login[24451]: segfault at 000000000000000c rip 0000003c7de610a2 rsp 00007fff07116968 error 4

I'm having a really hard time getting a core dump

Yeah, it's difficult to get login processes to core dump. In v1.2 it's easier though. But there's an alternative way to get the backtrace:

First set login_process_per_connection=no. Then:

gdb -p pidof imap-login cont <wait for crash> bt full

Tim, Thanks for the feedback. In the other email you sent about re-producing with nessus, note that we're using the checkpassword system, however from strace info so far we think the error happens before any fork happens to the custon auth program.

Your suggestion for trapping with gdb worked for me! Though I had to do this in gdb first cause I kept getting sigpipe breaks: handle SIGPIPE nostop noprint pass

This is my stack trace without debug symbols. How much would symbols help you here?

Program received signal SIGSEGV, Segmentation fault. 0x0000003c7de610a2 in krb5_is_referral_realm () from /usr/lib64/libkrb5.so.3 (gdb) bt full #0 0x0000003c7de610a2 in krb5_is_referral_realm () from /usr/lib64/libkrb5.so.3 No symbol table info available. #1 0x0000003c7de48ade in krb5_kt_get_entry () from /usr/lib64/libkrb5.so.3 No symbol table info available. #2 0x0000003c7fe3871e in kssl_keytab_is_available () from /lib64/libssl.so.6 No symbol table info available. #3 0x0000003c7fe1e345 in ssl3_choose_cipher () from /lib64/libssl.so.6 No symbol table info available. #4 0x0000003c7fe19aeb in ssl3_get_client_hello () from /lib64/libssl.so.6 No symbol table info available. #5 0x0000003c7fe1a465 in ssl3_accept () from /lib64/libssl.so.6 No symbol table info available. #6 0x0000003c7fe22602 in ssl23_get_client_hello () from /lib64/libssl.so.6 No symbol table info available. #7 0x0000003c7fe22d99 in ssl23_accept () from /lib64/libssl.so.6 No symbol table info available. #8 0x00000000004093f9 in ssl_step () No symbol table info available. #9 0x00000000004095e4 in ssl_proxy_new () No symbol table info available. #10 0x00000000004073b7 in login_accept_ssl () No symbol table info available. #11 0x0000000000411dc8 in io_loop_handler_run () No symbol table info available. #12 0x0000000000410edd in io_loop_run () No symbol table info available. #13 0x000000000040706e in main () No symbol table info available.

On Feb 22, 2010, at 11:57 AM, Timo Sirainen wrote:

Well, that's coming from Kerberos library, which is called by OpenSSL for some reason.. Are you using Kerberos? Anyway it looks to me more like OpenSSL or Kerberos bug.

Tim,

Below is the stack trace with symbols. The bug appears to manifest only in 64bit redhat/centos 5 only but happens against multiple versions of openssl that existed over 5's life. Unfortunately, RedHat decided to compile in kerberos so I can't control that. We played around but couldn't find a way to make it stop by manipulating ssl_cipher_list.

I have seen dovecot crash when the following packages are installed: openssl-0.9.8e-12.el5, openssl-0.9.8e-12.el5_4.1

I've reduced the test case to this:

31705 (SSL Cipher Suites Supported) - http://www.nessus.org/plugins/index.php?view=single&id=21643 When run manually from command line, I had to replace 443 with 993 or 995 inside the ssl_supported_ciphers.nasl script. Then I can just run this to make it happen: nasl -t ssl_supported_ciphers.nasl

While this is clearly an openssl bug, I cannot reproduce this on courier, but I did find a reference to a similar backtrace with stunnel: http://tinyurl.com/yeyo7t9

Can you think of any way I could disable kerberos for dovecot so this does not segfault? Is there any check we could put in the code to prevent the segfault?

Thanks, Todd

Program received signal SIGSEGV, Segmentation fault. 0x0000003adf4610a2 in krb5_is_referral_realm () from /usr/lib64/libkrb5.so.3 (gdb) bt full #0 0x0000003adf4610a2 in krb5_is_referral_realm () from /usr/lib64/libkrb5.so.3 No symbol table info available. #1 0x0000003adf448ade in krb5_kt_get_entry () from /usr/lib64/libkrb5.so.3 No symbol table info available. #2 0x0000003ae083876e in kssl_keytab_is_available () from /lib64/libssl.so.6 No symbol table info available. #3 0x0000003ae081e385 in ssl3_choose_cipher () from /lib64/libssl.so.6 No symbol table info available. #4 0x0000003ae0819b2b in ssl3_get_client_hello () from /lib64/libssl.so.6 No symbol table info available. #5 0x0000003ae081a4a5 in ssl3_accept () from /lib64/libssl.so.6 No symbol table info available. #6 0x0000003ae0822642 in ssl23_get_client_hello () from /lib64/libssl.so.6 No symbol table info available. #7 0x0000003ae0822dd9 in ssl23_accept () from /lib64/libssl.so.6 No symbol table info available. #8 0x000000000040a8b2 in ssl_handshake (proxy=0x1a793920) at ssl-proxy-openssl.c:399 ret = 0 #9 0x000000000040ab50 in ssl_step (proxy=0x1a793920) at ssl-proxy-openssl.c:456 No locals. #10 0x0000000000417927 in io_loop_handler_run (ioloop=0x1a789d70) at ioloop-epoll.c:209 ctx = (struct ioloop_handler_context *) 0x1a78bf00 events = (struct epoll_event *) 0x1a78d670 event = (const struct epoll_event *) 0x1a78d670 list = (struct io_list *) 0x1a7907f0 io = (struct io_file *) 0x1a795e50 tv = {tv_sec = 179, tv_usec = 999415} events_count = 7 t_id = 2 msecs = 180000 ret = 1 i = 0 j = 0 call = true #11 0x0000000000416b32 in io_loop_run (ioloop=0x1a789d70) at ioloop.c:336 No locals. #12 0x0000000000408dbd in main (argc=1, argv=0x7fffeae55498, envp=0x7fffeae554a8) at main.c:482

On Mon, 2010-03-01 at 11:48 -0600, Todd Rinaldo wrote:

I opened a bug with Red Hat on this issue. Someone just commented in the ticket that the issue is probably related to chroot. Does this put things back in the dovecot court? A full stack trace with symbols is in the ticket now.

You can try setting login_chroot=no to see if it goes away. But I don't think a library should be crashing when running on chroot environment..

https://bugzilla.redhat.com/show_bug.cgi?id=567711

That's not publicly visible.