Perhaps for whose interested - IETF RFC 7027 specifies for TLS use:
[ brainpoolP256r1 | brainpoolP384r1 | brainpoolP512r1 ]
And thus t1 would not work anyway. However, having tested r1 the result was just the same.
A tcpdump during the openssl test [ s_server | s_client ] then revealed (TLSv1.2 Record Layer: Handshake Protocol: Client Hello) :
Extension: supported_groups (len=10) Type: supported_groups (10) Length: 10 Supported Groups List Length: 8 Supported Groups (4 groups) Supported Group: x25519 (0x001d) Supported Group: secp256r1 (0x0017) Supported Group: secp521r1 (0x0019) Supported Group: secp384r1 (0x0018)
Apparently [ brainpool ] would apparently not fit into any of those groups. Perhaps a bug in OpenSSL 1.1.0h thus.
Turned out not being a bug in OpenSSL after all. From the cli it works with no issues this way:
[ openssl s_server -cert ec.cert.pem -key ec.key.pem -port 5555 -curves brainpoolP512r1 ] [ openssl s_client -connect localhost:5555 -curves brainpoolP512r1 ]
I am not familiar really with the OpenSSL API and only roughly gather that the app (dovecot) would have to make the API call [ SSL_CTX_set1_groups_list ] (https://www.openssl.org/docs/manmaster/man3/SSL_CTX_set1_groups.html) in order to support those curves.
Whoops.
We have a setting called
ssl_curve_listin dovecot, and I tried using that when I was testing. Turns out that there is a bug preventing that setting from being used. If you are compiling yourself, you can use the attached patch to fix this.After applying, you can set
ssl_curve_list = brainpoolP512r1
And then you can connect again.
Aki
Meantime I stumbled over that setting and was like 'yeah - what are you blubbering about when dovecot caters for it already'. That stopped when testing the setting ... like you said it is a bug apparently.
Now about compiling... that is not really my turf unless it is absolutely necessary. Time being I will (have to) work around with [ ssl_alt_key/cert ] and will notify the downstream repo maintainer about the patch, assuming that needs all that compiling I cannot just modify some file manually.