Is there any way, with postfix, to run a script on authentication failure, with information like the IP address and the username passed, for instance.
I wrote my own script for omprog which is part of rsyslog to read dovecot auth fails. Because postfix authenticates with dovecot and dovecot will log the auth failures.
If you don't know how omprog works, in concept it is "reading the logs", but in practice it gets loaded in memory by rsyslog and is feed log lines to parse in real time instead of continuously launching a new script instance to read over existing log entries.
My script parses the IP, username, and password sha1 hash to decide if the IP should be added to ipset. The password hash lets me know if someone is repeatedly trying the same password vs running through a dictionary attack.