Crashing when run against OpenSSL 1.1.0c
Hi,
I am running Dovecot 2.2.26.0 compiled against OpenSSL 1.1 and, since upgrading to OpenSSL 1.1.0c, the "lmtp" process has been crashing with SIGSEGV whenever it receives SIGINT. This always happens a minute or so after the lmtp process handles a message. It can also be manually reproduced by sending SIGINT to one of the running lmtp processes.
I am compiling and running on an Ubuntu 17.04 x86_64 system using GCC 6.2.
Here is the output of me reproducing it with gdb:
(gdb) signal SIGINT Continuing with signal SIGINT.
Program received signal SIGSEGV, Segmentation fault. 0x00007f6748cc2fb0 in ?? () (gdb) bt #0 0x00007f6748cc2fb0 in ?? () #1 0x00007f674872ac60 in ossl_init_thread_stop (locals=<optimized out>) at crypto/init.c:336 #2 0x00007f674872aee4 in OPENSSL_cleanup () at crypto/init.c:391 #3 0x00007f67491052e0 in __run_exit_handlers (status=0, listp=0x7f674948c5d8 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at exit.c:83 #4 0x00007f674910533a in __GI_exit (status=<optimized out>) at exit.c:105 #5 0x00007f67490eb3f8 in __libc_start_main (main=0x555b35fbfbc0 <main>, argc=1, argv=0x7ffd4ede3588, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffd4ede3578) at ../csu/libc-start.c:325 #6 0x0000555b35fbfe3a in _start ()
Here is the output of "doveconf -n":
# 2.2.26.0 (23d1de6): /etc/dovecot/dovecot.conf # Pigeonhole version 0.4.devel (623ae77) # OS: Linux 4.8.7-040807-generic x86_64 Ubuntu Zesty Zapus (development branch) auth_mechanisms = plain login auth_username_format = %Ln mail_location = mdbox:~/mdbox mailbox_list_index = yes managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { driver = pam } plugin { sieve = file:~/sieve;active=~/.dovecot.sieve } postmaster_address = michael@michaelmarley.com protocols = imap sieve lmtp service auth { client_limit = 1624 unix_listener /var/spool/postfix/private/dovecot-auth { group = postfix mode = 0660 user = postfix } } service imap-login { inet_listener imaps { port = 0 } } service lmtp { process_min_avail = 5 unix_listener /var/spool/postfix/private/dovecot-lmtp { group = postfix mode = 0600 user = postfix } } ssl = required ssl_ca =
I tried recompiling Dovecot to see if that might do any good, but I got the same result. Also, I noticed that the "test-crypto" test in src/lib-dcrypt also crashes on exit (despite the fact that all the tests succeeded) with a very similar stacktrace:
(gdb) run Starting program: /home/michael/Source/dovecot/core/src/lib-dcrypt/test-crypto [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". test_cipher_test_vectors ............................................. : ok test_cipher_aead_test_vectors ........................................ : ok test_hmac_test_vectors ............................................... : ok test_load_v1_keys .................................................... : ok test_load_v1_key ..................................................... : ok test_load_v1_public_key .............................................. : ok test_load_v2_key ..................................................... : ok test_load_v2_public_key .............................................. : ok test_get_info_v2_key ................................................. : ok test_gen_and_get_info_rsa_pem ........................................ : ok test_get_info_rsa_private_key ........................................ : ok test_get_info_invalid_keys ........................................... : ok test_get_info_key_encrypted .......................................... : ok test_get_info_pw_encrypted ........................................... : ok test_password_change ................................................. : ok test_load_invalid_keys ............................................... : ok 0 / 16 tests failed
Program received signal SIGSEGV, Segmentation fault. 0x00007ffff75a5480 in ?? () (gdb) bt #0 0x00007ffff75a5480 in ?? () #1 0x00007ffff6ff7c60 in ossl_init_thread_stop (locals=<optimized out>) at crypto/init.c:336 #2 0x00007ffff6ff7ee4 in OPENSSL_cleanup () at crypto/init.c:391 #3 0x00007ffff78472e0 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 #4 0x00007ffff784733a in exit () from /lib/x86_64-linux-gnu/libc.so.6 #5 0x00007ffff782d3f8 in __libc_start_main () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x000055555555f68a in _start ()
If there is anything I have missed or if there is any other way I can help, please let me know.
Thanks,
Michael Marley
On 15.11.2016 13:27, Michael Marley wrote:
Hi,
I am running Dovecot 2.2.26.0 compiled against OpenSSL 1.1 and, since upgrading to OpenSSL 1.1.0c, the "lmtp" process has been crashing with SIGSEGV whenever it receives SIGINT. This always happens a minute or so after the lmtp process handles a message. It can also be manually reproduced by sending SIGINT to one of the running lmtp processes.
I am compiling and running on an Ubuntu 17.04 x86_64 system using GCC 6.2.
Here is the output of me reproducing it with gdb:
(gdb) signal SIGINT Continuing with signal SIGINT.
Program received signal SIGSEGV, Segmentation fault. 0x00007f6748cc2fb0 in ?? () (gdb) bt #0 0x00007f6748cc2fb0 in ?? () #1 0x00007f674872ac60 in ossl_init_thread_stop (locals=<optimized out>) at crypto/init.c:336 #2 0x00007f674872aee4 in OPENSSL_cleanup () at crypto/init.c:391 #3 0x00007f67491052e0 in __run_exit_handlers (status=0, listp=0x7f674948c5d8 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at exit.c:83 #4 0x00007f674910533a in __GI_exit (status=<optimized out>) at exit.c:105 #5 0x00007f67490eb3f8 in __libc_start_main (main=0x555b35fbfbc0 <main>, argc=1, argv=0x7ffd4ede3588, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffd4ede3578) at ../csu/libc-start.c:325 #6 0x0000555b35fbfe3a in _start ()
Here is the output of "doveconf -n":
# 2.2.26.0 (23d1de6): /etc/dovecot/dovecot.conf # Pigeonhole version 0.4.devel (623ae77) # OS: Linux 4.8.7-040807-generic x86_64 Ubuntu Zesty Zapus (development branch) auth_mechanisms = plain login auth_username_format = %Ln mail_location = mdbox:~/mdbox mailbox_list_index = yes managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { driver = pam } plugin { sieve = file:~/sieve;active=~/.dovecot.sieve } postmaster_address = michael@michaelmarley.com protocols = imap sieve lmtp service auth { client_limit = 1624 unix_listener /var/spool/postfix/private/dovecot-auth { group = postfix mode = 0660 user = postfix } } service imap-login { inet_listener imaps { port = 0 } } service lmtp { process_min_avail = 5 unix_listener /var/spool/postfix/private/dovecot-lmtp { group = postfix mode = 0600 user = postfix } } ssl = required ssl_ca =
I tried recompiling Dovecot to see if that might do any good, but I got the same result. Also, I noticed that the "test-crypto" test in src/lib-dcrypt also crashes on exit (despite the fact that all the tests succeeded) with a very similar stacktrace:
(gdb) run Starting program: /home/michael/Source/dovecot/core/src/lib-dcrypt/test-crypto [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". test_cipher_test_vectors ............................................. : ok test_cipher_aead_test_vectors ........................................ : ok test_hmac_test_vectors ............................................... : ok test_load_v1_keys .................................................... : ok test_load_v1_key ..................................................... : ok test_load_v1_public_key .............................................. : ok test_load_v2_key ..................................................... : ok test_load_v2_public_key .............................................. : ok test_get_info_v2_key ................................................. : ok test_gen_and_get_info_rsa_pem ........................................ : ok test_get_info_rsa_private_key ........................................ : ok test_get_info_invalid_keys ........................................... : ok test_get_info_key_encrypted .......................................... : ok test_get_info_pw_encrypted ........................................... : ok test_password_change ................................................. : ok test_load_invalid_keys ............................................... : ok 0 / 16 tests failed
Program received signal SIGSEGV, Segmentation fault. 0x00007ffff75a5480 in ?? () (gdb) bt #0 0x00007ffff75a5480 in ?? () #1 0x00007ffff6ff7c60 in ossl_init_thread_stop (locals=<optimized out>) at crypto/init.c:336 #2 0x00007ffff6ff7ee4 in OPENSSL_cleanup () at crypto/init.c:391 #3 0x00007ffff78472e0 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 #4 0x00007ffff784733a in exit () from /lib/x86_64-linux-gnu/libc.so.6 #5 0x00007ffff782d3f8 in __libc_start_main () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x000055555555f68a in _start ()
If there is anything I have missed or if there is any other way I can help, please let me know.
Thanks,
Michael Marley Hi!
This is a known problem, and we have received patch for it which we are reviewing. At the moment Dovecot is not officially compatible with OpenSSL 1.1.0.
Aki
Hi
You can't think how glad I am that SSL issues rise again in a new Dovecot version with next Ubuntu release with a new OpenSSL library. Some days ago I have posted something similar about Ubuntu 14.04 - Dovecot 2.2.9 - OpenSSL 1.0 (Dovecot processes turning zombie) but noone cared about. I still think is somehow related to ssl-param process + config + auth + ...whatever (all of them "ignoring idle SIGINT")
If Dovecot SSL implementation is so dependant of a certain version of a library (OpenSSL for example) you should consider saving a copy of the "known-good" library version somewhere in Dovecot private space and use it without relaing on generic system upgrades.
Don't get me wrong: I love Dovecot as IMAP server and local delivery agent. But public interface is unreliable, authentication too, so for now I am using Dovecot as an isolated server in localhost and attach other public interfaces to it. Even so, delivery agent LDA is still trying to authenticate and complains about null passwords (what?). I have destination addresses that should go to a shared mailbox and that user@domain is never allowed to login. The workaround is to set an "impossible" password for those but this is not a nice solution. LDA should care only about what counts for him (maybe using some defaults) and leave everything else for the "big boys".
Sorry for any inconvenience, Adrian POPA
----- Original Message ----- From: "Aki Tuomi" aki.tuomi@dovecot.fi To: dovecot@dovecot.org Sent: Tuesday, November 15, 2016 1:43 PM Subject: Re: Crashing when run against OpenSSL 1.1.0c
On 15.11.2016 13:27, Michael Marley wrote:
Hi,
I am running Dovecot 2.2.26.0 compiled against OpenSSL 1.1 and, since upgrading to OpenSSL 1.1.0c, the "lmtp" process has been crashing with SIGSEGV whenever it receives SIGINT. This always happens a minute or so after the lmtp process handles a message. It can also be manually reproduced by sending SIGINT to one of the running lmtp processes.
I am compiling and running on an Ubuntu 17.04 x86_64 system using GCC 6.2.
Here is the output of me reproducing it with gdb:
(gdb) signal SIGINT Continuing with signal SIGINT.
Program received signal SIGSEGV, Segmentation fault. 0x00007f6748cc2fb0 in ?? () (gdb) bt #0 0x00007f6748cc2fb0 in ?? () #1 0x00007f674872ac60 in ossl_init_thread_stop (locals=<optimized out>) at crypto/init.c:336 #2 0x00007f674872aee4 in OPENSSL_cleanup () at crypto/init.c:391 #3 0x00007f67491052e0 in __run_exit_handlers (status=0, listp=0x7f674948c5d8 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at exit.c:83 #4 0x00007f674910533a in __GI_exit (status=<optimized out>) at exit.c:105 #5 0x00007f67490eb3f8 in __libc_start_main (main=0x555b35fbfbc0 <main>, argc=1, argv=0x7ffd4ede3588, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffd4ede3578) at ../csu/libc-start.c:325 #6 0x0000555b35fbfe3a in _start ()
Here is the output of "doveconf -n":
# 2.2.26.0 (23d1de6): /etc/dovecot/dovecot.conf # Pigeonhole version 0.4.devel (623ae77) # OS: Linux 4.8.7-040807-generic x86_64 Ubuntu Zesty Zapus (development branch) auth_mechanisms = plain login auth_username_format = %Ln mail_location = mdbox:~/mdbox mailbox_list_index = yes managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { driver = pam } plugin { sieve = file:~/sieve;active=~/.dovecot.sieve } postmaster_address = michael@michaelmarley.com protocols = imap sieve lmtp service auth { client_limit = 1624 unix_listener /var/spool/postfix/private/dovecot-auth { group = postfix mode = 0660 user = postfix } } service imap-login { inet_listener imaps { port = 0 } } service lmtp { process_min_avail = 5 unix_listener /var/spool/postfix/private/dovecot-lmtp { group = postfix mode = 0600 user = postfix } } ssl = required ssl_ca =
I tried recompiling Dovecot to see if that might do any good, but I got the same result. Also, I noticed that the "test-crypto" test in src/lib-dcrypt also crashes on exit (despite the fact that all the tests succeeded) with a very similar stacktrace:
(gdb) run Starting program: /home/michael/Source/dovecot/core/src/lib-dcrypt/test-crypto [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". test_cipher_test_vectors ............................................. : ok test_cipher_aead_test_vectors ........................................ : ok test_hmac_test_vectors ............................................... : ok test_load_v1_keys .................................................... : ok test_load_v1_key ..................................................... : ok test_load_v1_public_key .............................................. : ok test_load_v2_key ..................................................... : ok test_load_v2_public_key .............................................. : ok test_get_info_v2_key ................................................. : ok test_gen_and_get_info_rsa_pem ........................................ : ok test_get_info_rsa_private_key ........................................ : ok test_get_info_invalid_keys ........................................... : ok test_get_info_key_encrypted .......................................... : ok test_get_info_pw_encrypted ........................................... : ok test_password_change ................................................. : ok test_load_invalid_keys ............................................... : ok 0 / 16 tests failed
Program received signal SIGSEGV, Segmentation fault. 0x00007ffff75a5480 in ?? () (gdb) bt #0 0x00007ffff75a5480 in ?? () #1 0x00007ffff6ff7c60 in ossl_init_thread_stop (locals=<optimized out>) at crypto/init.c:336 #2 0x00007ffff6ff7ee4 in OPENSSL_cleanup () at crypto/init.c:391 #3 0x00007ffff78472e0 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 #4 0x00007ffff784733a in exit () from /lib/x86_64-linux-gnu/libc.so.6 #5 0x00007ffff782d3f8 in __libc_start_main () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x000055555555f68a in _start ()
If there is anything I have missed or if there is any other way I can help, please let me know.
Thanks,
Michael Marley Hi!
This is a known problem, and we have received patch for it which we are reviewing. At the moment Dovecot is not officially compatible with OpenSSL 1.1.0.
Aki
On November 15, 2016 at 7:42 PM Adrian POPA adrianp@aageneral.ro wrote:
Hi
You can't think how glad I am that SSL issues rise again in a new Dovecot version with next Ubuntu release with a new OpenSSL library. Some days ago I have posted something similar about Ubuntu 14.04 - Dovecot 2.2.9 - OpenSSL 1.0 (Dovecot processes turning zombie) but noone cared about. I still think is somehow related to ssl-param process + config + auth + ...whatever (all of them "ignoring idle SIGINT")
Well, 2.2.9 is pretty old. It was released almost 4 years ago.
If Dovecot SSL implementation is so dependant of a certain version of a library (OpenSSL for example) you should consider saving a copy of the "known-good" library version somewhere in Dovecot private space and use it without relaing on generic system upgrades.
OpenSSL has breaking API changes between 1.0.0, 1.0.1, 1.0.2 and 1.1.0.
Don't get me wrong: I love Dovecot as IMAP server and local delivery agent. But public interface is unreliable, authentication too, so for now I am using Dovecot as an isolated server in localhost and attach other public interfaces to it. Even so, delivery agent LDA is still trying to authenticate and complains about null passwords (what?). I have destination addresses that should go to a shared mailbox and that user@domain is never allowed to login. The workaround is to set an "impossible" password for those but this is not a nice solution. LDA should care only about what counts for him (maybe using some defaults) and leave everything else for the "big boys".
Perhaps you could post your doveconf -n to some new thread along with some explanation of your setup and problem you are trying to solve.
Sorry for any inconvenience, Adrian POPA
Aki Tuomi Dovecot oy
participants (3)
-
Adrian POPA
-
Aki Tuomi
-
Michael Marley