poolmon improvements

[Timo Sirainen]

I've been planning to improve poolmon failure checking for a long time already, but I still haven't managed to get to it. Maybe somebody else has more time, so here's a feature request for anyone to implement:

poolmon currently gives up immediately if the first check to any service fails. It really should be trying multiple times over multiple seconds before giving up. I think ideally it would be:

 - Individual check timeout could still be the default 5 seconds
 - Add full check time setting, which could be e.g. 15 seconds. If all checks fail during this time then disable the host.
 - If request fails because connection gets rejected, retry quickly, e.g. after 0,1 seconds
 - If check fails because of protocol errors, wait for a long time, e.g. 1 second

So this would avoid backend being removed in situations where it really shouldn't be removed:

 - Dovecot restarts
 - Dovecot reloads
 - load spikes and other random issues that cause temporary problems

Especially the load spike is an annoying issue which my plan doesn't even fully solve. The solution to fix a heavily overloaded cluster isn't really to start removing all of its backends that are busy working..