I did some local testing and it seems that you are using a curve that is not acceptable for openssl as a server key.
I tested with openssl s_server -cert ec-cert.pem -key ec-key.pem -port 5555
using cert generated with brainpool. Everything works if I use prime256v1 or secp521r1. This is a limitation in OpenSSL and not something we can really do anything about.
Aki Tuomi Open-Xchange Oy Which openssl version you are using? This end it is OpenSSL 1.1.0h. There are no issues creating private keys, issuing csr, signing certs with that particular curve. Printing certs and verifying certs against keys is panning out too, comparing md5 hashes also no errors. So why would openssl not accept (limit) keys is has generated and verified with no error?
Ran both certificate types with [ openssl s_server -cert ec.cert.pem -key ec.key.pem -port 5555 ] and [ openssl s_server -cert rsa.cert.pem -key rsa.key.pem -port 5555 ] and both with the output:
Using default temp DH parameters ACCEPT
Which would indicate this not being caused by openssl.