TLS communication director -> backend with X.509 cert checks?
Hello,
using Dovecot 2.2.9 and a setup with directors and backends. The communication between directors and backends needs to be TLS secured.
The director config contains a list of hostnames for the backends. (implicit list because of multiple A/AAAA records for a single hostname or explicit list of several host names)
On connection setup from a client the director connects to the selected backend. But it seems (not checked in the source yet), that for SSL certificate verification the director doesn't know the original host name anymore. The certificate's CN gets compared to the IP address the director connects to.
Oct 12 23:56:51 director2 dovecot: director: Error: director(2001:683:921:f33::5:1:9090/out): connect() failed: Connection reset by peer
Oct 12 23:57:53 director2 dovecot: imap-login: Error: proxy: hostname doesn't match SSL certificate at 2001:683:921:f33::5:fe:993: user=<foo>, method=PLAIN, rip=2001:638:912:f33::1:1, lip=2001:638:912:f33::5:2, TLS, session=
In 10-directors.conf I've: director_mail_servers = backend1.<domain> backend2.<domain>
Should I create certificates with IP address in SAN? (Any hint about the correct syntax for the openssl.conf is welcome). Or is there any chance that this is fixed already or will be fixed in the near future or even better, that it's my fault?
Best regards from Dresden/Germany
Viele Grüße aus Dresden
Heiko Schlittermann-- SCHLITTERMANN.de ---------------------------- internet & unix support - Heiko Schlittermann, Dipl.-Ing. (TU) - {fon,fax}: +49.351.802998{1,3} - gnupg encrypted messages are welcome --------------- key ID: F69376CE - ! key id 7CBF764A and 972EAC9F are revoked since 2015-01 ------------ -
On 13 Oct 2015, at 21:44, Heiko Schlittermann hs@schlittermann.de wrote:
Hello,
using Dovecot 2.2.9 and a setup with directors and backends. The communication between directors and backends needs to be TLS secured.
The director config contains a list of hostnames for the backends. (implicit list because of multiple A/AAAA records for a single hostname or explicit list of several host names)
On connection setup from a client the director connects to the selected backend. But it seems (not checked in the source yet), that for SSL certificate verification the director doesn't know the original host name anymore. The certificate's CN gets compared to the IP address the director connects to.
Right. The hostnames are lost immediately at director startup. I've never really thought about needing this functionality for director, since they're usually in the same trusted network with backends..
Should I create certificates with IP address in SAN? (Any hint about the correct syntax for the openssl.conf is welcome). Or is there any chance that this is fixed already or will be fixed in the near future or even better, that it's my fault?
I guess that could work for now. No idea about how to do such certificates.
Timo Sirainen tss@iki.fi (Di 13 Okt 2015 21:02:59 CEST): …
On connection setup from a client the director connects to the selected backend. But it seems (not checked in the source yet), that for SSL certificate verification the director doesn't know the original host name anymore. The certificate's CN gets compared to the IP address the director connects to.
Right. The hostnames are lost immediately at director startup. I've never really thought about needing this functionality for director, since they're usually in the same trusted network with backends..
That's it… "ususally". And additionally local policy says that we should use secured connections whenever credentials are transported … And since the director uses either a master password or the credentials obtained from the client, we want to use secured connections. And using TLS w/o verified certs is better than nothing, but it's far from being perfect.
I see:
a) pass the host *names* to the director too, for CN verification
purpose
May be in struct mail_host could be a field for the original
hostname we used to obtain the adress(es)?or b) allow some kind of certificate pinning, that is loose the implied relation CN <=> hostname
Should I create certificates with IP address in SAN? (Any hint about the correct syntax for the openssl.conf is welcome). Or is there any chance that this is fixed already or will be fixed in the near future or even better, that it's my fault?
I guess that could work for now. No idea about how to do such certificates.
I'll try that, but I think it's not a solution as soon as we reach out for "official" certs. And because it puts more details about the infrastructure into the configuration than would be necessary.
Best regards from Dresden/Germany
Viele Grüße aus Dresden
Heiko Schlittermann-- SCHLITTERMANN.de ---------------------------- internet & unix support - Heiko Schlittermann, Dipl.-Ing. (TU) - {fon,fax}: +49.351.802998{1,3} - gnupg encrypted messages are welcome --------------- key ID: F69376CE - ! key id 7CBF764A and 972EAC9F are revoked since 2015-01 ------------ -
On 13 Oct 2015, at 22:18, Heiko Schlittermann hs@schlittermann.de wrote:
Timo Sirainen tss@iki.fi (Di 13 Okt 2015 21:02:59 CEST): …
On connection setup from a client the director connects to the selected backend. But it seems (not checked in the source yet), that for SSL certificate verification the director doesn't know the original host name anymore. The certificate's CN gets compared to the IP address the director connects to.
Right. The hostnames are lost immediately at director startup. I've never really thought about needing this functionality for director, since they're usually in the same trusted network with backends..
That's it… "ususally". And additionally local policy says that we should use secured connections whenever credentials are transported … And since the director uses either a master password or the credentials obtained from the client, we want to use secured connections. And using TLS w/o verified certs is better than nothing, but it's far from being perfect.
I've been planning to add support for non-plaintext SASL for Dovecot proxy. Probably SCRAM-SHA1. That would avoid sending credentials in plaintext, although it wouldn't prevent other kind of MITM.
I see:
a) pass the host *names* to the director too, for CN verification purpose
May be in struct mail_host could be a field for the original hostname we used to obtain the adress(es)?
Does the attached patch work? Compiles, but untested.
Timo Sirainen tss@iki.fi (Di 13 Okt 2015 21:36:40 CEST): …
I see:
a) pass the host *names* to the director too, for CN verification purpose
May be in struct mail_host could be a field for the original hostname we used to obtain the adress(es)?Does the attached patch work? Compiles, but untested.
I'm about to test it.
Best regards from Dresden/Germany
Viele Grüße aus Dresden
Heiko Schlittermann-- SCHLITTERMANN.de ---------------------------- internet & unix support - Heiko Schlittermann, Dipl.-Ing. (TU) - {fon,fax}: +49.351.802998{1,3} - gnupg encrypted messages are welcome --------------- key ID: F69376CE - ! key id 7CBF764A and 972EAC9F are revoked since 2015-01 ------------ -
Hi Timo,
Heiko Schlittermann hs@schlittermann.de (Di 13 Okt 2015 22:33:23 CEST):
Does the attached patch work? Compiles, but untested. I'm about to test it.
It seems to update the struct mail_host, but it looks as if the data in mail_host do not propagate down to login_proxy_new().
In other words, in login_proxy_new() set->host contains the IP address, correctly, because the director choose it, but where can I find the hostname there? And we need a way to pass the host*name* further, to the SSL verifcation step, don't we?
Best regards from Dresden/Germany
Viele Grüße aus Dresden
Heiko Schlittermann-- SCHLITTERMANN.de ---------------------------- internet & unix support - Heiko Schlittermann, Dipl.-Ing. (TU) - {fon,fax}: +49.351.802998{1,3} - gnupg encrypted messages are welcome --------------- key ID: F69376CE - ! key id 7CBF764A and 972EAC9F are revoked since 2015-01 ------------ -
On 14 Oct 2015, at 00:34, Heiko Schlittermann hs@schlittermann.de wrote:
Hi Timo,
Heiko Schlittermann hs@schlittermann.de (Di 13 Okt 2015 22:33:23 CEST):
Does the attached patch work? Compiles, but untested. I'm about to test it.
It seems to update the struct mail_host, but it looks as if the data in mail_host do not propagate down to login_proxy_new().
In other words, in login_proxy_new() set->host contains the IP address, correctly, because the director choose it, but where can I find the hostname there? And we need a way to pass the host*name* further, to the SSL verifcation step, don't we?
Proxying in general does check that hostname matches the SSL certificate, because both the hostname and IP address are sent to login process. So it should work in a way that host=<hostname> and hostip=<ip> is sent. I thought my patch did that.. Normally auth_debug=yes would be enough to debug this, but this happens between director and login process so I don't think it's going to be of much use. login process's client_auth_parse_args() is what should see these two parameters correctly.
I can check this further tomorrow.
Timo Sirainen tss@iki.fi (Di 13 Okt 2015 23:49:20 CEST): …
Proxying in general does check that hostname matches the SSL certificate, because both the hostname and IP address are sent to login process. So it should work in a way that host=<hostname> and hostip=<ip> is sent. I thought my patch did that.. Normally auth_debug=yes would be enough to debug this, but this happens between director and login process so I don't think it's going to be of much use. login process's client_auth_parse_args() is what should see these two parameters correctly.
I can check this further tomorrow.
I've put an i_warning("*** %s: ...", __FUNCTION__, ...) into several places.
Oct 14 00:02:33 director1 dovecot: director: Warning: *** login_host_callback: OK#0112#011user=foo#011proxy#011ssl=yes#011nopassword=y#011lip=2001:x.y:f33::5:1#011lport=993#011pass=x#011proxy_refresh=450#011host=2001:x.y:f33::5:fe
Here it seems that the director doesn't send it's knowledge about the hostname.
Here some other output, to show that the host list contains names and addresses:
Oct 14 00:02:32 director1 dovecot: director: Warning: ** mail_host_add: added backends.<domain> [2001:x.y:f33::5:fe] Oct 14 00:02:32 director1 dovecot: director: Warning: ** mail_host_add: added backends.<domain> [2001:x.y:f33::5:ff] Oct 14 00:02:32 director1 dovecot: director: Warning: ** mail_host_add: added backends.<domain> [149.x.y.103] Oct 14 00:02:32 director1 dovecot: director: Warning: ** mail_host_add: added backends.<domain> [149.x.y.102]
-- Heiko
Heiko Schlittermann hs@schlittermann.de (Mi 14 Okt 2015 00:10:50 CEST):
Timo Sirainen tss@iki.fi (Di 13 Okt 2015 23:49:20 CEST): …
Proxying in general does check that hostname matches the SSL certificate, because both the hostname and IP address are sent to login process. So it should work in a way that host=<hostname> and hostip=<ip> is sent. I thought my patch did that.. Normally auth_debug=yes would be enough to debug this, but this happens between director and login process so I don't think it's going to be of much use. login process's client_auth_parse_args() is what should see these two parameters correctly.
I can check this further tomorrow.
I've put an i_warning("*** %s: ...", __FUNCTION__, ...) into several places.
Oct 14 00:02:33 director1 dovecot: director: Warning: *** login_host_callback: OK#0112#011user=foo#011proxy#011ssl=yes#011nopassword=y#011lip=2001:x.y:f33::5:1#011lport=993#011pass=x#011proxy_refresh=450#011host=2001:x.y:f33::5:fe
Here it seems that the director doesn't send it's knowledge about the hostname.
Here some other output, to show that the host list contains names and addresses:
Oct 14 00:02:32 director1 dovecot: director: Warning: ** mail_host_add: added backends.<domain> [2001:x.y:f33::5:fe] Oct 14 00:02:32 director1 dovecot: director: Warning: ** mail_host_add: added backends.<domain> [2001:x.y:f33::5:ff] Oct 14 00:02:32 director1 dovecot: director: Warning: ** mail_host_add: added backends.<domain> [149.x.y.103] Oct 14 00:02:32 director1 dovecot: director: Warning: ** mail_host_add: added backends.<domain> [149.x.y.102]
And if I add -D to the director service, I can see "Debug: request <hash> refreshed timeout to …", but never I see "Debug: request <hash> added". And from what I understand this would be the place where the mail_host info comes into the game.
But probably I do not understand how director_request_continue() is supposed to work.
Best regards from Dresden/Germany
Viele Grüße aus Dresden
Heiko Schlittermann-- SCHLITTERMANN.de ---------------------------- internet & unix support - Heiko Schlittermann, Dipl.-Ing. (TU) - {fon,fax}: +49.351.802998{1,3} - gnupg encrypted messages are welcome --------------- key ID: F69376CE - ! key id 7CBF764A and 972EAC9F are revoked since 2015-01 ------------ -
Heiko Schlittermann hs@schlittermann.de (Mi 14 Okt 2015 00:46:11 CEST): …
And if I add -D to the director service, I can see "Debug: request <hash> refreshed timeout to …", but never I see "Debug: request <hash> added". And from what I understand this would be the place where the mail_host info comes into the game.
But probably I do not understand how director_request_continue() is supposed to work.
Ah, the information comes from the other director running. The other one is using an unpatched version of dovecot.
If I shutdown the other director instance, it seems to work. Tomorrow I'll do more testing. Good work, thank you.
BTW: I've put there an IPv6 address into the director_servers list (not an DNS name).
director_servers = 2001:x:y:f33::5:1
…
inet_listener {
address = ::
port = 9090
}it doesn't recognize itself: Oct 14 01:06:13 director1 dovecot: director: Fatal: director_servers doesn't list ourself
director_servers = 2001:x:y:f33::5:1:9090
…
inet_listener {
address = ::
port = 9090
}works, but is ambigous, isn't it? Shouldn't we use [2001:x:y:f33::5:1]:9090 in such a case? But: *Unknown director host: [2001:x:y:f33::5:1]*
Best regards from Dresden/Germany
Viele Grüße aus Dresden
Heiko Schlittermann-- SCHLITTERMANN.de ---------------------------- internet & unix support - Heiko Schlittermann, Dipl.-Ing. (TU) - {fon,fax}: +49.351.802998{1,3} - gnupg encrypted messages are welcome --------------- key ID: F69376CE - ! key id 7CBF764A and 972EAC9F are revoked since 2015-01 ------------ -
Hi Timo
Heiko Schlittermann hs@schlittermann.de (Mi 14 Okt 2015 01:10:20 CEST): …
Ah, the information comes from the other director running. The other one is using an unpatched version of dovecot.
Your patch for backend-certificate verification works. Thank you for the good and fast work. Is there any chance that this will make it into Dovecot's next release?
BTW: The ambiguity of 2001:db8::9090 remains. Shouldn't you allow [2001:db8::]¹ resp [2001:db8::9090]¹ resp. [2001:db8::]:9090² for such cases? (In case one want's to use IPv6 addresses instead of hostnames in the director_servers option. (And probably in other places too.))
¹) Address ²) Address:Port
Best regards from Dresden/Germany
Viele Grüße aus Dresden
Heiko Schlittermann-- SCHLITTERMANN.de ---------------------------- internet & unix support - Heiko Schlittermann, Dipl.-Ing. (TU) - {fon,fax}: +49.351.802998{1,3} - gnupg encrypted messages are welcome --------------- key ID: F69376CE - ! key id 7CBF764A and 972EAC9F are revoked since 2015-01 ------------ -
On 15 Oct 2015, at 00:28, Heiko Schlittermann hs@schlittermann.de wrote:
Hi Timo
Heiko Schlittermann hs@schlittermann.de (Mi 14 Okt 2015 01:10:20 CEST): …
Ah, the information comes from the other director running. The other one is using an unpatched version of dovecot.
Your patch for backend-certificate verification works. Thank you for the good and fast work. Is there any chance that this will make it into Dovecot's next release?
Implemented also support for sending the hostname within director ring:
http://hg.dovecot.org/dovecot-2.2/rev/8e9cada0c8fc http://hg.dovecot.org/dovecot-2.2/rev/7f718c840aff http://hg.dovecot.org/dovecot-2.2/rev/5876ca2d63fb
Although it's not possible right now to add hostname using "doveadm director add", so that probably needs to be implemented at some point.
BTW: The ambiguity of 2001:db8::9090 remains. Shouldn't you allow [2001:db8::]¹ resp [2001:db8::9090]¹ resp. [2001:db8::]:9090² for such cases? (In case one want's to use IPv6 addresses instead of hostnames in the director_servers option. (And probably in other places too.))
Timo Sirainen tss@iki.fi (Di 13 Okt 2015 21:02:59 CEST):
the IP address the director connects to.
Right. The hostnames are lost immediately at director startup. I've never really thought about needing this functionality for director, since they're usually in the same trusted network with backends..
Ooo. What if
director_mail_servers = backends.<domain>and the DNS entry for backends.<domain> gets updated? Does the director catch up the change automatically w/o restart?
Best regards from Dresden/Germany
Viele Grüße aus Dresden
Heiko Schlittermann-- SCHLITTERMANN.de ---------------------------- internet & unix support - Heiko Schlittermann, Dipl.-Ing. (TU) - {fon,fax}: +49.351.802998{1,3} - gnupg encrypted messages are welcome --------------- key ID: F69376CE - ! key id 7CBF764A and 972EAC9F are revoked since 2015-01 ------------ -
On 13 Oct 2015, at 22:21, Heiko Schlittermann hs@schlittermann.de wrote:
Timo Sirainen tss@iki.fi (Di 13 Okt 2015 21:02:59 CEST):
the IP address the director connects to.
Right. The hostnames are lost immediately at director startup. I've never really thought about needing this functionality for director, since they're usually in the same trusted network with backends..
Ooo. What if
director_mail_servers = backends.<domain>
and the DNS entry for backends.<domain> gets updated? Does the director catch up the change automatically w/o restart?
No, and I'm not sure it even should. Use "doveadm director ring remove" to get rid of unwanted directors.
participants (2)
-
Heiko Schlittermann
-
Timo Sirainen