I did some local testing and it seems that you are using a curve that is not acceptable for openssl as a server key. I tested with openssl s_server -cert ec-cert.pem -key ec-key.pem -port 5555 using cert generated with brainpool. Everything works if I use prime256v1 or secp521r1. This is a limitation in OpenSSL and not something we can really do anything about. Aki Tuomi Open-Xchange Oy Which openssl version you are using? This end it is OpenSSL 1.1.0h. There are no issues creating private keys, issuing csr, signing certs with that particular curve. Printing certs and verifying certs against keys is panning out too, comparing md5 hashes also no errors. So why would openssl not accept (limit) keys is has generated and verified with no error?
try
openssl s_server -cert /path/to/cert -key /path/to/key -port 5555
openssl s_client -connect localhost:5555
Uhum, I see now. What a strange thing (bug?) openssl is doing. Thank you for valuable time/effort having debug this. Seems I have to start the CA all over...
Perhaps for whose interested - IETF RFC 7027 specifies for TLS use:
[ brainpoolP256r1 | brainpoolP384r1 | brainpoolP512r1 ]
And thus t1 would not work anyway. However, having tested r1 the result was just the same.
A tcpdump during the openssl test [ s_server | s_client ] then revealed (TLSv1.2 Record Layer: Handshake Protocol: Client Hello) :
Extension: supported_groups (len=10) Type: supported_groups (10) Length: 10 Supported Groups List Length: 8 Supported Groups (4 groups) Supported Group: x25519 (0x001d) Supported Group: secp256r1 (0x0017) Supported Group: secp521r1 (0x0019) Supported Group: secp384r1 (0x0018)
Apparently [ brainpool ] would apparently not fit into any of those groups. Perhaps a bug in OpenSSL 1.1.0h thus.