AV> So i’m using dovecot, and i created a self signed certificate AV> with mkcert.sh based on dovecot-openssl.cnf. The name in there matches my mail server.
AV> The first time it connects in mac mail however, it says the AV> certificate is invalid and another server might pretend to be me etc.
AV> I then have the option of trusting it.
AV> Is this normal behaviour? Will it always be invalid if it’s not signed by a third party?
Yes. The point of a trusted CA signing your cert is that they have steps to "verify" who you are and that you're "authorized" to issue certs for the listed FQDNs. Without that, ANYONE could create a cert, and sign it and then present it to people connecting to your mail server [perhaps using a MITM style attack.] The connecting party would have no way to tell if your cert vs the attackers cert was actually valid.
It would be like showing up at the bank and having this exchange:
You: "Hey, I'm Jim Bob - can I take money out of his account?" Bank: "Do you have some ID?" You: "Yeah! See, I have this plastic card with my picture and name, that I ginned up in the basement."
Now does the bank say: "Yeah, that looks fine." or do they say "You know we really need ID [a certificate] that's authenticated and issued [signed] by the state [third-party/trusted CA.]."
I think it's obvious that accepting your basement produced ID would be a problem. [Even if we also admit that while the state issued ID (or trusted CA signed certs) has some additional value, it isn't without potential flaws, etc.]
The alternative would be to add your CA cert [the one you signed the server cert with] to all the connecting clients as a trusted CA. This way your self signed cert would now be "trusted."
[The details are left as an exercise to the reader. Google is your friend.]
-Greg