[Dovecot] 2.1.2 (pop3|imap)-login crash
Hi Timo, hi all, after upgrading my server (both backends and load balancer) to 2.1.2 (from 2.0.17), I'm getting a log of login processes crashed in load balancer.
Log lines are like:
Mar 20 10:05:45 mailgw-lb dovecot: pop3-login: Fatal: master: service(pop3-login): child 27764 killed with signal 11 (core dumps disabled) Mar 20 10:06:17 mailgw-lb dovecot: imap-login: Fatal: master: service(imap-login): child 28468 killed with signal 11 (core dumps disabled)
Stack trace of a crashed process is like:
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread -1220163904 (LWP 27764)]
0xb77cd176 in ssl_proxy_is_handshaked (proxy=0x0) at ssl-proxy-openssl.c:710
710 {
(gdb) bt
#0 0xb77cd176 in ssl_proxy_is_handshaked (proxy=0x0) at
ssl-proxy-openssl.c:710
#1 0xb77c7295 in client_get_log_str (client=0x807b830, msg=0x804e290
"proxy(aaaaaa@dddddd.it): disconnecting x.x.x.x (Disconnected by
server)") at client-common.c:469
#2 0xb77c73c6 in client_log (client=0x807b830, msg=0x804e290
"proxy(aaaaaaa@dddddd.it): disconnecting x.x.x.x (Disconnected by
server)") at client-common.c:553
#3 0xb77c9a45 in login_proxy_free_reason (_proxy=<value optimized out>,
reason=0x804e248 "Disconnected by server") at login-proxy.c:373
#4 0xb77ca9b5 in server_input (proxy=0x0) at login-proxy.c:93
#5 0xb7793762 in io_loop_call_io (io=0x8094180) at ioloop.c:380
#6 0xb7794cc9 in io_loop_handler_run (ioloop=0x8055480) at
ioloop-epoll.c:213
#7 0xb77936f9 in io_loop_run (ioloop=0x8055480) at ioloop.c:399
#8 0xb777e4c8 in master_service_run (service=0x80553b0,
callback=0xb77cc110
The strange part of the story is that not all process crash. I'm trying to figure out if only TLS/SSL process crash.
Any idea?
Thanks
Luca
Hi,
On 20.3.2012, at 11.09, Luca Palazzo wrote:
Hi Timo, hi all, after upgrading my server (both backends and load balancer) to 2.1.2 (from 2.0.17), I'm getting a log of login processes crashed in load balancer.
0xb77cd176 in ssl_proxy_is_handshaked (proxy=0x0) at ssl-proxy-openssl.c:710 710 { (gdb) bt #0 0xb77cd176 in ssl_proxy_is_handshaked (proxy=0x0) at ssl-proxy-openssl.c:710 #1 0xb77c7295 in client_get_log_str (client=0x807b830, msg=0x804e290 "proxy(aaaaaa@dddddd.it): disconnecting x.x.x.x (Disconnected by server)") at client-common.c:469 #2 0xb77c73c6 in client_log (client=0x807b830, msg=0x804e290 "proxy(aaaaaaa@dddddd.it): disconnecting x.x.x.x (Disconnected by server)") at client-common.c:553 #3 0xb77c9a45 in login_proxy_free_reason (_proxy=<value optimized out>, reason=0x804e248 "Disconnected by server") at login-proxy.c:373
Interesting. This happens because client_destroy() has already been called at the time login_proxy_free_reason() gets called. I'll need to look further into it, but for a quick workaround use the attached patch.
It worked. We have no more sigsegv on *-login process.
Thanks
Luca
Nella citazione in data Wed Mar 21 16:17:56 2012, Timo Sirainen ha scritto:
Hi,
On 20.3.2012, at 11.09, Luca Palazzo wrote:
Hi Timo, hi all, after upgrading my server (both backends and load balancer) to 2.1.2 (from 2.0.17), I'm getting a log of login processes crashed in load balancer.
0xb77cd176 in ssl_proxy_is_handshaked (proxy=0x0) at ssl-proxy-openssl.c:710 710 { (gdb) bt #0 0xb77cd176 in ssl_proxy_is_handshaked (proxy=0x0) at ssl-proxy-openssl.c:710 #1 0xb77c7295 in client_get_log_str (client=0x807b830, msg=0x804e290 "proxy(aaaaaa@dddddd.it): disconnecting x.x.x.x (Disconnected by server)") at client-common.c:469 #2 0xb77c73c6 in client_log (client=0x807b830, msg=0x804e290 "proxy(aaaaaaa@dddddd.it): disconnecting x.x.x.x (Disconnected by server)") at client-common.c:553 #3 0xb77c9a45 in login_proxy_free_reason (_proxy=<value optimized out>, reason=0x804e248 "Disconnected by server") at login-proxy.c:373
Interesting. This happens because client_destroy() has already been called at the time login_proxy_free_reason() gets called. I'll need to look further into it, but for a quick workaround use the attached patch.
The log messages are now wrong though. It logs SSL/TLS connections as being non-SSL/TLS. Oh, right, this must have started happening because of this recent change: http://hg.dovecot.org/dovecot-2.1/rev/49b832c5de0e
I'll figure out a proper fix soon.
On 21.3.2012, at 18.04, Luca Palazzo wrote:
It worked. We have no more sigsegv on *-login process.
Thanks
Luca
Nella citazione in data Wed Mar 21 16:17:56 2012, Timo Sirainen ha scritto:
Hi,
On 20.3.2012, at 11.09, Luca Palazzo wrote:
Hi Timo, hi all, after upgrading my server (both backends and load balancer) to 2.1.2 (from 2.0.17), I'm getting a log of login processes crashed in load balancer.
0xb77cd176 in ssl_proxy_is_handshaked (proxy=0x0) at ssl-proxy-openssl.c:710 710 { (gdb) bt #0 0xb77cd176 in ssl_proxy_is_handshaked (proxy=0x0) at ssl-proxy-openssl.c:710 #1 0xb77c7295 in client_get_log_str (client=0x807b830, msg=0x804e290 "proxy(aaaaaa@dddddd.it): disconnecting x.x.x.x (Disconnected by server)") at client-common.c:469 #2 0xb77c73c6 in client_log (client=0x807b830, msg=0x804e290 "proxy(aaaaaaa@dddddd.it): disconnecting x.x.x.x (Disconnected by server)") at client-common.c:553 #3 0xb77c9a45 in login_proxy_free_reason (_proxy=<value optimized out>, reason=0x804e248 "Disconnected by server") at login-proxy.c:373
Interesting. This happens because client_destroy() has already been called at the time login_proxy_free_reason() gets called. I'll need to look further into it, but for a quick workaround use the attached patch.
Hi,
These should fix it properly:
http://hg.dovecot.org/dovecot-2.1/rev/1d23440ccb89 http://hg.dovecot.org/dovecot-2.1/rev/842e5124038d
On Wed, 2012-03-21 at 18:08 +0200, Timo Sirainen wrote:
The log messages are now wrong though. It logs SSL/TLS connections as being non-SSL/TLS. Oh, right, this must have started happening because of this recent change: http://hg.dovecot.org/dovecot-2.1/rev/49b832c5de0e
I'll figure out a proper fix soon.
On 21.3.2012, at 18.04, Luca Palazzo wrote:
It worked. We have no more sigsegv on *-login process.
Thanks
Luca
Nella citazione in data Wed Mar 21 16:17:56 2012, Timo Sirainen ha scritto:
Hi,
On 20.3.2012, at 11.09, Luca Palazzo wrote:
Hi Timo, hi all, after upgrading my server (both backends and load balancer) to 2.1.2 (from 2.0.17), I'm getting a log of login processes crashed in load balancer.
0xb77cd176 in ssl_proxy_is_handshaked (proxy=0x0) at ssl-proxy-openssl.c:710 710 { (gdb) bt #0 0xb77cd176 in ssl_proxy_is_handshaked (proxy=0x0) at ssl-proxy-openssl.c:710 #1 0xb77c7295 in client_get_log_str (client=0x807b830, msg=0x804e290 "proxy(aaaaaa@dddddd.it): disconnecting x.x.x.x (Disconnected by server)") at client-common.c:469 #2 0xb77c73c6 in client_log (client=0x807b830, msg=0x804e290 "proxy(aaaaaaa@dddddd.it): disconnecting x.x.x.x (Disconnected by server)") at client-common.c:553 #3 0xb77c9a45 in login_proxy_free_reason (_proxy=<value optimized out>, reason=0x804e248 "Disconnected by server") at login-proxy.c:373
Interesting. This happens because client_destroy() has already been called at the time login_proxy_free_reason() gets called. I'll need to look further into it, but for a quick workaround use the attached patch.
I've applied both and reverted previous one. Everything seems to run flowless. Thanks Luca
On 03/22/2012 03:05 PM, Timo Sirainen wrote:
Hi,
These should fix it properly:
http://hg.dovecot.org/dovecot-2.1/rev/1d23440ccb89 http://hg.dovecot.org/dovecot-2.1/rev/842e5124038d
On Wed, 2012-03-21 at 18:08 +0200, Timo Sirainen wrote:
The log messages are now wrong though. It logs SSL/TLS connections as being non-SSL/TLS. Oh, right, this must have started happening because of this recent change: http://hg.dovecot.org/dovecot-2.1/rev/49b832c5de0e
I'll figure out a proper fix soon.
On 21.3.2012, at 18.04, Luca Palazzo wrote:
It worked. We have no more sigsegv on *-login process.
Thanks
Luca
Nella citazione in data Wed Mar 21 16:17:56 2012, Timo Sirainen ha scritto:
Hi,
On 20.3.2012, at 11.09, Luca Palazzo wrote:
Hi Timo, hi all, after upgrading my server (both backends and load balancer) to 2.1.2 (from 2.0.17), I'm getting a log of login processes crashed in load balancer.
0xb77cd176 in ssl_proxy_is_handshaked (proxy=0x0) at ssl-proxy-openssl.c:710 710 { (gdb) bt #0 0xb77cd176 in ssl_proxy_is_handshaked (proxy=0x0) at ssl-proxy-openssl.c:710 #1 0xb77c7295 in client_get_log_str (client=0x807b830, msg=0x804e290 "proxy(aaaaaa@dddddd.it): disconnecting x.x.x.x (Disconnected by server)") at client-common.c:469 #2 0xb77c73c6 in client_log (client=0x807b830, msg=0x804e290 "proxy(aaaaaaa@dddddd.it): disconnecting x.x.x.x (Disconnected by server)") at client-common.c:553 #3 0xb77c9a45 in login_proxy_free_reason (_proxy=<value optimized out>, reason=0x804e248 "Disconnected by server") at login-proxy.c:373
Interesting. This happens because client_destroy() has already been called at the time login_proxy_free_reason() gets called. I'll need to look further into it, but for a quick workaround use the attached patch.
participants (2)
-
Luca Palazzo
-
Timo Sirainen