Tom put forth on 11/9/2010 8:53 PM:
we have recently had some brute force attacks on the pop3 and imapd and this results in many processes being used for login attempts.
Our dovecot is hosted on a Virtual Private Server which restricts access to IPTABLEs and also make a limit on the number of processes that can be running
So I can't restrict the attackers IP addresses via IPTABLES, as we don't have access to that. I can't really patch dovecot as we are reliant on the distro packages.
Dovecot isn't iptables. It is an IMAP/POP3 server. It implements basic user account security. Preventing DOS or other attacks is not its job. That is the job of the kernel. There are many reasons why applications don't duplicate kernel functionality, and most should be obvious to anyone who thinks on the matter for a few moments. I'm not going to bother listing them here.
You went cheap and/or didn't research the provider/features, and now are feeling the sting. Find a new VPS provider, or upgrade to one of their packages that allows access to iptables so you can run fail2ban. I don't think you're going to be able to make headway here with Dovecot.
Some lessons are, unfortunately, learned best the hard way. :(
-- Stan