Perhaps there are dovecot (and postfix submission) options to at least restrict access by IP?
Restricting by IP is soon going to become very tedious, especially if you are dealing with more than a small number of users, and especially once post-COVID travel comes back and people start connecting from random hotels and airport lounges.
If you don't fancy the idea of client certs, the alternative I would suggest instead of IP limiting would be a Wireguard VPN instead of IP limiting.
Wireguard VPN servers run very quiet and won't respond to anything unless a client sends the right parameters.
Of course the downside of a VPN compared to certificates is that the user will have to be aware and know how to manage a VPN, whilst with certificates it can all be quietly done in the background.