Perhaps for whose interested - IETF RFC 7027 specifies for TLS use:
[ brainpoolP256r1 | brainpoolP384r1 | brainpoolP512r1 ]
And thus t1 would not work anyway. However, having tested r1 the result was just the same.
A tcpdump during the openssl test [ s_server | s_client ] then revealed (TLSv1.2 Record Layer: Handshake Protocol: Client Hello) :
Extension: supported_groups (len=10) Type: supported_groups (10) Length: 10 Supported Groups List Length: 8 Supported Groups (4 groups) Supported Group: x25519 (0x001d) Supported Group: secp256r1 (0x0017) Supported Group: secp521r1 (0x0019) Supported Group: secp384r1 (0x0018)
Apparently [ brainpool ] would apparently not fit into any of those groups. Perhaps a bug in OpenSSL 1.1.0h thus.
Turned out not being a bug in OpenSSL after all. From the cli it works with no issues this way:
[ openssl s_server -cert ec.cert.pem -key ec.key.pem -port 5555 -curves brainpoolP512r1 ] [ openssl s_client -connect localhost:5555 -curves brainpoolP512r1 ]
I am not familiar really with the OpenSSL API and only roughly gather that the app (dovecot) would have to make the API call [ SSL_CTX_set1_groups_list ] (https://www.openssl.org/docs/manmaster/man3/SSL_CTX_set1_groups.html) in order to support those curves.