[Dovecot] [bug] ssl-params hangs when FIPS is enabled

[Michal Hlavinka]

Hi,

we found a bug in ssl-params. It calls openssl DH generator for 512 and 
1024 bits, but in FIPS mode, openssl won't generate anything for less 
than 1024, so it fails with:
error:0506A06E:Diffie-Hellman routines:DH_BUILTIN_GENPARAMS:key size too 
small

but when DH generator fails, ssl-params hangs forever in io_loop_run:
__epoll_wait_nocancel()
io_loop_handler_run(..) at ioloop-epoll.c:176
io_loop_run(..) at ioloop.c:406
master_service_run(..) at master-service.c:566
main(..) at main.c:156

Getting system in FIPS mode can be a little tricky, but DH_generator 
simulated failure can do the trick. Just change 
ssl-params/ssl-params-openssl.c:generate_dh_paramaters:39
-       if (dh == NULL) {
+       if (1 || dh == NULL) {

and when you run ssl-params (with no ssl-parameters.dat), it hangs.

Let me know if you need more information.

Regards,
Michal Hlavinka