Strange "IMAP connection broken (server response)" errors
Hello,
We upgraded one of our dovecot servers to debian stretch with dovecot 2.2.27 and since then one of our users has been experiencing random IMAP failures.
We enabled raw logging at the server side and it shows normal IMAP commands/responses:
1507292522.222427 * 6 FETCH (FLAGS () BODYSTRUCTURE ("text" "plain" ("charset" "us-ascii") NIL NIL "7bit" 4645 112 NIL NIL NIL NIL)) 1507292522.222653 00000011 OK Fetch completed (0.006 + 0.000 secs).
On the client side the user runs alpine and the corresponding debug lines:
IMAP DEBUG 14:22:02.216167: 00000011 FETCH 6 (BODYSTRUCTURE FLAGS)
14:22:02.217396 IMAP 14:22:02 10/6 mm_notify bye: {[127.0.0.1]:1555/imap/user="ha4aa"}INBOX: [CLOSED] IMAP connection broken (server response)
14:22:02.217471 IMAP 14:22:02 10/6 mm_log error: [CLOSED] IMAP connection broken (server response)
The "[127.0.0.1]:1555/imap/user="ha4aa" part in the log comes from an socat inserted between the client and the server to check independetly the imap session. According to socat, the server response didn't reach the client!:
00000010 OK Fetch completed (0.005 + 0.000 secs).\r
2017/10/06 14:22:02.216299 length=40 from=845 to=884 00000011 FETCH 6 (BODYSTRUCTURE FLAGS)\r
and here ends the socat log, the server response didn't reach the client.
The same alpine package is used against older dovecot servers from debian jessie and works flawlessly.
The server and client are both KVM guests and there's no any content filtering software between them. There's no corresponding error in the dovecot log file and dovecot didn't regard the connection broken and detected client close:
... Oct 6 14:19:44 mail2 dovecot: imap(ha4aa): Debug: Mailbox INBOX: Opened mail UID=55244 because: MIME part Oct 6 14:22:02 mail2 dovecot: imap(ha4aa): Connection closed in=797 out=73151
We are out of the ideas what can be the reason for the random lost server responses.
Any help is appreciated!
Best regards, Jozsef
E-mail : kadlecsik.jozsef@wigner.mta.hu PGP key: http://www.kfki.hu/~kadlec/pgp_public_key.txt Address: Wigner Research Centre for Physics, Hungarian Academy of Sciences H-1525 Budapest 114, POB. 49, Hungary
Hello,
On Fri, 6 Oct 2017, Jozsef Kadlecsik wrote:
We upgraded one of our dovecot servers to debian stretch with dovecot 2.2.27 and since then one of our users has been experiencing random IMAP failures.
We enabled raw logging at the server side and it shows normal IMAP commands/responses:
1507292522.222427 * 6 FETCH (FLAGS () BODYSTRUCTURE ("text" "plain" ("charset" "us-ascii") NIL NIL "7bit" 4645 112 NIL NIL NIL NIL)) 1507292522.222653 00000011 OK Fetch completed (0.006 + 0.000 secs).
On the client side the user runs alpine and the corresponding debug lines:
IMAP DEBUG 14:22:02.216167: 00000011 FETCH 6 (BODYSTRUCTURE FLAGS)
14:22:02.217396 IMAP 14:22:02 10/6 mm_notify bye: {[127.0.0.1]:1555/imap/user="ha4aa"}INBOX: [CLOSED] IMAP connection broken (server response)
14:22:02.217471 IMAP 14:22:02 10/6 mm_log error: [CLOSED] IMAP connection broken (server response)
The "[127.0.0.1]:1555/imap/user="ha4aa" part in the log comes from an socat inserted between the client and the server to check independently the imap session. According to socat, the server response didn't reach the client!:
00000010 OK Fetch completed (0.005 + 0.000 secs).\r
2017/10/06 14:22:02.216299 length=40 from=845 to=884 00000011 FETCH 6 (BODYSTRUCTURE FLAGS)\r
and here ends the socat log.
The date of the last rawlog line corresponds to an ssl debug log of dovecot (from the last run):
Oct 20 18:50:05 mail2 dovecot: imap-login: Debug: SSL error: SSL_read() failed: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I added the patch "openssl: Clear error queue after an incomplete SSL_shutdown", recompiled the package but it did not help.
Any help is highly welcomed!
Best regards, Jozsef
E-mail : kadlecsik.jozsef@wigner.mta.hu PGP key: http://www.kfki.hu/~kadlec/pgp_public_key.txt Address: Wigner Research Centre for Physics, Hungarian Academy of Sciences H-1525 Budapest 114, POB. 49, Hungary
On Fri, 20 Oct 2017, Kadlecsik József wrote:
On Fri, 6 Oct 2017, Jozsef Kadlecsik wrote:
We upgraded one of our dovecot servers to debian stretch with dovecot 2.2.27 and since then one of our users has been experiencing random IMAP failures.
On the client side the user runs alpine and the corresponding debug lines:
IMAP DEBUG 14:22:02.216167: 00000011 FETCH 6 (BODYSTRUCTURE FLAGS)
14:22:02.217396 IMAP 14:22:02 10/6 mm_notify bye: {[127.0.0.1]:1555/imap/user="ha4aa"}INBOX: [CLOSED] IMAP connection broken (server response)
The date of the last rawlog line corresponds to an ssl debug log of dovecot (from the last run):
Oct 20 18:50:05 mail2 dovecot: imap-login: Debug: SSL error: SSL_read() failed: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
It is an openssl compatibility issue introduced in OpenSSL 1.0.2f. The IMAP failures could be solved with the following patches, which are similar to what nginx uses (http://hg.nginx.org/nginx/rev/062c189fee20): For Dovecot 2.2.35: diff --git a/src/lib-ssl-iostream/iostream-openssl.c b/src/lib-ssl-iostream/iostream-openssl.c index 68ec221..31d1017 100644 --- a/src/lib-ssl-iostream/iostream-openssl.c +++ b/src/lib-ssl-iostream/iostream-openssl.c @@ -324,7 +324,7 @@ static void openssl_iostream_unref(struct ssl_iostream *ssl_io) static void openssl_iostream_destroy(struct ssl_iostream *ssl_io) { - if (SSL_shutdown(ssl_io->ssl) != 1) { + if (!SSL_in_init(ssl_io->ssl) && SSL_shutdown(ssl_io->ssl) != 1) { /* if bidirectional shutdown fails we need to clear the error queue */ openssl_iostream_clear_errors(); diff --git a/src/login-common/ssl-proxy-openssl.c b/src/login-common/ssl-proxy-openssl.c index 947c8ef..3ac6823 100644 --- a/src/login-common/ssl-proxy-openssl.c +++ b/src/login-common/ssl-proxy-openssl.c @@ -833,7 +833,7 @@ void ssl_proxy_destroy(struct ssl_proxy *proxy) if (proxy->io_plain_write != NULL) io_remove(&proxy->io_plain_write); - if (SSL_shutdown(proxy->ssl) != 1) { + if (!SSL_in_init(proxy->ssl) && SSL_shutdown(proxy->ssl) != 1) { /* if bidirectional shutdown fails we need to clear the error queue. */ openssl_iostream_clear_errors(); For Dovecot master branch: diff --git a/src/lib-ssl-iostream/iostream-openssl.c b/src/lib-ssl-iostream/iostream-openssl.c index 45de412..ed1f0a4 100644 --- a/src/lib-ssl-iostream/iostream-openssl.c +++ b/src/lib-ssl-iostream/iostream-openssl.c @@ -345,7 +345,7 @@ static void openssl_iostream_unref(struct ssl_iostream *ssl_io) static void openssl_iostream_destroy(struct ssl_iostream *ssl_io) { - if (SSL_shutdown(ssl_io->ssl) != 1) { + if (!SSL_in_init(ssl_io->ssl) && SSL_shutdown(ssl_io->ssl) != 1) { /* if bidirectional shutdown fails we need to clear the error queue */ openssl_iostream_clear_errors(); Best regards, Jozsef -- E-mail : kadlecsik.jozsef@wigner.mta.hu PGP key: http://www.kfki.hu/~kadlec/pgp_public_key.txt Address: Wigner Research Centre for Physics, Hungarian Academy of Sciences H-1525 Budapest 114, POB. 49, Hungary
On 20.03.2018 10:16, Kadlecsik József wrote:
On Fri, 20 Oct 2017, Kadlecsik József wrote:
On Fri, 6 Oct 2017, Jozsef Kadlecsik wrote:
We upgraded one of our dovecot servers to debian stretch with dovecot 2.2.27 and since then one of our users has been experiencing random IMAP failures.
On the client side the user runs alpine and the corresponding debug lines:
IMAP DEBUG 14:22:02.216167: 00000011 FETCH 6 (BODYSTRUCTURE FLAGS)
14:22:02.217396 IMAP 14:22:02 10/6 mm_notify bye: {[127.0.0.1]:1555/imap/user="ha4aa"}INBOX: [CLOSED] IMAP connection broken (server response) The date of the last rawlog line corresponds to an ssl debug log of dovecot (from the last run):
Oct 20 18:50:05 mail2 dovecot: imap-login: Debug: SSL error: SSL_read() failed: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init It is an openssl compatibility issue introduced in OpenSSL 1.0.2f. The IMAP failures could be solved with the following patches, which are similar to what nginx uses (http://hg.nginx.org/nginx/rev/062c189fee20):
For Dovecot 2.2.35:
diff --git a/src/lib-ssl-iostream/iostream-openssl.c b/src/lib-ssl-iostream/iostream-openssl.c index 68ec221..31d1017 100644 --- a/src/lib-ssl-iostream/iostream-openssl.c +++ b/src/lib-ssl-iostream/iostream-openssl.c @@ -324,7 +324,7 @@ static void openssl_iostream_unref(struct ssl_iostream *ssl_io)
static void openssl_iostream_destroy(struct ssl_iostream *ssl_io) { - if (SSL_shutdown(ssl_io->ssl) != 1) { + if (!SSL_in_init(ssl_io->ssl) && SSL_shutdown(ssl_io->ssl) != 1) { /* if bidirectional shutdown fails we need to clear the error queue */ openssl_iostream_clear_errors(); diff --git a/src/login-common/ssl-proxy-openssl.c b/src/login-common/ssl-proxy-openssl.c index 947c8ef..3ac6823 100644 --- a/src/login-common/ssl-proxy-openssl.c +++ b/src/login-common/ssl-proxy-openssl.c @@ -833,7 +833,7 @@ void ssl_proxy_destroy(struct ssl_proxy *proxy) if (proxy->io_plain_write != NULL) io_remove(&proxy->io_plain_write);
- if (SSL_shutdown(proxy->ssl) != 1) { + if (!SSL_in_init(proxy->ssl) && SSL_shutdown(proxy->ssl) != 1) { /* if bidirectional shutdown fails we need to clear the error queue. */ openssl_iostream_clear_errors();
For Dovecot master branch:
diff --git a/src/lib-ssl-iostream/iostream-openssl.c b/src/lib-ssl-iostream/iostream-openssl.c index 45de412..ed1f0a4 100644 --- a/src/lib-ssl-iostream/iostream-openssl.c +++ b/src/lib-ssl-iostream/iostream-openssl.c @@ -345,7 +345,7 @@ static void openssl_iostream_unref(struct ssl_iostream *ssl_io)
static void openssl_iostream_destroy(struct ssl_iostream *ssl_io) { - if (SSL_shutdown(ssl_io->ssl) != 1) { + if (!SSL_in_init(ssl_io->ssl) && SSL_shutdown(ssl_io->ssl) != 1) { /* if bidirectional shutdown fails we need to clear the error queue */ openssl_iostream_clear_errors();
Best regards, Jozsef -- E-mail : kadlecsik.jozsef@wigner.mta.hu PGP key: http://www.kfki.hu/~kadlec/pgp_public_key.txt Address: Wigner Research Centre for Physics, Hungarian Academy of Sciences H-1525 Budapest 114, POB. 49, Hungary Hi!
Thank you for your patch, we'll look into it. Aki
participants (3)
-
Aki Tuomi
-
Jozsef Kadlecsik
-
Kadlecsik József