Hello,
We’re rolling out large SNI deployments for our mail servers. Each domain gets an entry like this in the config:local_name mail.foo.com { ssl_cert = </ssl/domain_tls/*.foo.com/combined ssl_key = </ssl/domain_tls/*.foo.com/combined }
There are a couple problems we’re finding with this approach:Dovecot wants to load everything at once, which has some machines taking up many GiB of memory just for Dovecot. Is there any way to defer loading of an SSL cert until a client actually requests it?
Any time we add or remove a domain, Dovecot’s SNI config matrix needs to be rebuilt. Is there a way to handle SNI requests dynamically via some sort of configuration plugin, so we wouldn’t need to rebuild the config on domain add/remove? I looked through the docs but couldn’t see a way to do this.
Thank you in advance!
-Felipe Gasper Mississauga, ON
On 11.11.2016 01:02, Felipe Gasper wrote:
Hello,
We’re rolling out large SNI deployments for our mail servers. Each domain gets an entry like this in the config:
local_name mail.foo.com { ssl_cert = </ssl/domain_tls/*.foo.com/combined ssl_key = </ssl/domain_tls/*.foo.com/combined }
There are a couple problems we’re finding with this approach:
Dovecot wants to load everything at once, which has some machines taking up many GiB of memory just for Dovecot. Is there any way to defer loading of an SSL cert until a client actually requests it?
Any time we add or remove a domain, Dovecot’s SNI config matrix needs to be rebuilt. Is there a way to handle SNI requests dynamically via some sort of configuration plugin, so we wouldn’t need to rebuild the config on domain add/remove? I looked through the docs but couldn’t see a way to do this.
Thank you in advance!
-Felipe Gasper Mississauga, ON
Unfortunately it's not possible now, it has been asked before though. We have this feature request in our list but cannot give any date when it would be available.
Aki Tuomi
Dovecot oy
On Friday 11 of November 2016, Felipe Gasper wrote:
Hello,
We’re rolling out large SNI deployments for our mail servers. Each domain gets an entry like this in the config:
local_name mail.foo.com { ssl_cert = </ssl/domain_tls/*.foo.com/combined ssl_key = </ssl/domain_tls/*.foo.com/combined }
Lack of glob/regexp support here is also a problem (for me). I could have 50% smaller config if local_name supported regexp matching, so it would be possible to do:
local_name ^(pop3|imap)\.foo\.com { ... }
or even with glob like *.foo.com matching.
There are a couple problems we’re finding with this approach:
- Dovecot wants to load everything at once, which has some machines taking up many GiB of memory just for Dovecot. Is there any way to defer loading of an SSL cert until a client actually requests it?
No - thread here http://www.dovecot.org/list/dovecot/2016-October/105855.html
Memory is one thing.
The other is that dovecot stops accepting clients when huge config reload happens (I guess it's a design problem since it makes no sense to do that in any case. Clients should be processed without gap using old config until new config is loaded and ready to go).
And third problem is that there is hardcoded 10s limit for reloading which in case thousands of certificates is way too short limit. Anyway if you hit that limit it's already lost case due to earlier problem.
- Any time we add or remove a domain, Dovecot’s SNI config matrix needs to be rebuilt. Is there a way to handle SNI requests dynamically via some sort of configuration plugin, so we wouldn’t need to rebuild the config on domain add/remove? I looked through the docs but couldn’t see a way to do this.
That's unavoidable for now :-(
Here we started analyzing maillog and put into dovecot config only these ssl certs for domains that are actually used with TLS. It's very ugly and short- sighted approach but hopefuly proper solution will be implemented by dovecot team before all people start to use TLS.
Thank you in advance!
-Felipe Gasper Mississauga, ON
-- Arkadiusz Miśkiewicz, arekm / ( maven.pl | pld-linux.org )
On November 11, 2016 at 12:22 PM Arkadiusz Miśkiewicz <arekm@maven.pl> wrote:
On Friday 11 of November 2016, Felipe Gasper wrote:
Hello,
We’re rolling out large SNI deployments for our mail servers. Each domain gets an entry like this in the config:
local_name mail.foo.com { ssl_cert = </ssl/domain_tls/*.foo.com/combined ssl_key = </ssl/domain_tls/*.foo.com/combined }
Lack of glob/regexp support here is also a problem (for me). I could have 50% smaller config if local_name supported regexp matching, so it would be possible to do:
local_name ^(pop3|imap)\.foo\.com { ... }
or even with glob like *.foo.com matching.
There are a couple problems we’re finding with this approach:
- Dovecot wants to load everything at once, which has some machines taking up many GiB of memory just for Dovecot. Is there any way to defer loading of an SSL cert until a client actually requests it?
No - thread here http://www.dovecot.org/list/dovecot/2016-October/105855.html
Memory is one thing.
The other is that dovecot stops accepting clients when huge config reload happens (I guess it's a design problem since it makes no sense to do that in any case. Clients should be processed without gap using old config until new config is loaded and ready to go).
And third problem is that there is hardcoded 10s limit for reloading which in case thousands of certificates is way too short limit. Anyway if you hit that limit it's already lost case due to earlier problem.
- Any time we add or remove a domain, Dovecot’s SNI config matrix needs to be rebuilt. Is there a way to handle SNI requests dynamically via some sort of configuration plugin, so we wouldn’t need to rebuild the config on domain add/remove? I looked through the docs but couldn’t see a way to do this.
That's unavoidable for now :-(
Here we started analyzing maillog and put into dovecot config only these ssl certs for domains that are actually used with TLS. It's very ugly and short- sighted approach but hopefuly proper solution will be implemented by dovecot team before all people start to use TLS.
Thank you in advance!
-Felipe Gasper Mississauga, ON
-- Arkadiusz Miśkiewicz, arekm / ( maven.pl | pld-linux.org )
Hi!
We are going to do some changes at some point how the certs are loaded and handled to alleviate this. The idea is not yet ripe, so I won't go into too much detail, but idea is to move the cert storage from protocol login processes to elsewhere.
The local_name matching can probably be fixed faster, it could use the same rules as matching cert names generally do.
Aki
On Nov 11, 2016, at 5:36 AM, Aki Tuomi <aki.tuomi@dovecot.fi> wrote:
Hi!
We are going to do some changes at some point how the certs are loaded and handled to alleviate this. The idea is not yet ripe, so I won't go into too much detail, but idea is to move the cert storage from protocol login processes to elsewhere.
In the other thread (http://www.dovecot.org/list/dovecot/2016-October/105855.html) there is mention of Exim. Exim actually allows a pretty flexible SNI-time cert load. Dovecot’s config doesn’t have the run-time variables that would allow this, but maybe there could be some sort of pluggable mechanism to show Dovecot where the cert for a given FQDN is?
It’d be great if a little bit of logic could “teach” Dovecot for each system, like:
sub certificate_path ($fqdn) { return "$WHERE_MY_CERTS_ARE/$fqdn.pem"; }
-FG
On 11.11.2016 12:22, Arkadiusz Miśkiewicz wrote:
Hello,
We’re rolling out large SNI deployments for our mail servers. Each domain gets an entry like this in the config:
local_name mail.foo.com { ssl_cert = </ssl/domain_tls/*.foo.com/combined ssl_key = </ssl/domain_tls/*.foo.com/combined } Lack of glob/regexp support here is also a problem (for me). I could have 50% smaller config if local_name supported regexp matching, so it would be
On Friday 11 of November 2016, Felipe Gasper wrote: possible to do:
local_name ^(pop3|imap)\.foo\.com { ... }
or even with glob like *.foo.com matching.
There are a couple problems we’re finding with this approach:
- Dovecot wants to load everything at once, which has some machines taking up many GiB of memory just for Dovecot. Is there any way to defer loading of an SSL cert until a client actually requests it? No - thread here http://www.dovecot.org/list/dovecot/2016-October/105855.html
Memory is one thing.
The other is that dovecot stops accepting clients when huge config reload happens (I guess it's a design problem since it makes no sense to do that in any case. Clients should be processed without gap using old config until new config is loaded and ready to go).
And third problem is that there is hardcoded 10s limit for reloading which in case thousands of certificates is way too short limit. Anyway if you hit that limit it's already lost case due to earlier problem.
- Any time we add or remove a domain, Dovecot’s SNI config matrix needs to be rebuilt. Is there a way to handle SNI requests dynamically via some sort of configuration plugin, so we wouldn’t need to rebuild the config on domain add/remove? I looked through the docs but couldn’t see a way to do this. That's unavoidable for now :-(
Here we started analyzing maillog and put into dovecot config only these ssl certs for domains that are actually used with TLS. It's very ugly and short- sighted approach but hopefuly proper solution will be implemented by dovecot team before all people start to use TLS.
Thank you in advance!
-Felipe Gasper Mississauga, ON
If you are interested in testing, please find patch attached that allows you to specify
local_name *.foo.bar { }
or
local_name *.*.foo.bar { }
so basically you can now use certificate name matching rules for local_name. It made most sense.
This should apply cleanly to 2.2.26.0.
Aki Tuomi Dovecot oy
On Friday 11 of November 2016, Aki Tuomi wrote:
If you are interested in testing, please find patch attached that allows you to specify
local_name *.foo.bar { }
or
local_name *.*.foo.bar { }
so basically you can now use certificate name matching rules for local_name. It made most sense.
Great! Seems to be working fine for my usage and makes my configs 50% smaller (which is gigantic improvement). Will do more testing though.
Thanks!
What about dovecot stopping processing new clients when reload is in progress problem - is it possible to make it behave better? To minimize (or avoid) "downtime".
How to reproduce - just create config file with 20 000 - 50 000 entries
local_name hostX....example.com { ssl_cert = </etc/certs/cert.pem ssl_key = </etc/certs/cert.pem }
where cert.pem contains some full chain (CA cert + intermediate + cert + key).
Start dovecot and then doveadm reload should take long time. Enough for noticing that dovecot stops processing clients.
Aki Tuomi Dovecot oy
-- Arkadiusz Miśkiewicz, arekm / ( maven.pl | pld-linux.org )
On 11.11.2016 19:17, Arkadiusz Miśkiewicz wrote:
On Friday 11 of November 2016, Aki Tuomi wrote:
If you are interested in testing, please find patch attached that allows you to specify
local_name *.foo.bar { }
or
local_name *.*.foo.bar { }
so basically you can now use certificate name matching rules for local_name. It made most sense. Great! Seems to be working fine for my usage and makes my configs 50% smaller (which is gigantic improvement). Will do more testing though.
Thanks!
What about dovecot stopping processing new clients when reload is in progress problem - is it possible to make it behave better? To minimize (or avoid) "downtime".
How to reproduce - just create config file with 20 000 - 50 000 entries
local_name hostX....example.com { ssl_cert = </etc/certs/cert.pem ssl_key = </etc/certs/cert.pem }
where cert.pem contains some full chain (CA cert + intermediate + cert + key).
Start dovecot and then doveadm reload should take long time. Enough for noticing that dovecot stops processing clients.
Aki Tuomi Dovecot oy
That is something that will happen later. Can't give any date, but it's in our internal tasklist.
Aki
On Friday 11 of November 2016, Aki Tuomi wrote:
On 11.11.2016 19:17, Arkadiusz Miśkiewicz wrote:
On Friday 11 of November 2016, Aki Tuomi wrote:
If you are interested in testing, please find patch attached that allows you to specify
local_name *.foo.bar { }
or
local_name *.*.foo.bar { }
so basically you can now use certificate name matching rules for local_name. It made most sense.
Great! Seems to be working fine for my usage and makes my configs 50% smaller (which is gigantic improvement). Will do more testing though.
Thanks!
What about dovecot stopping processing new clients when reload is in progress problem - is it possible to make it behave better? To minimize (or avoid) "downtime".
How to reproduce - just create config file with 20 000 - 50 000 entries
local_name hostX....example.com {
ssl_cert = </etc/certs/cert.pem ssl_key = </etc/certs/cert.pem
}
where cert.pem contains some full chain (CA cert + intermediate + cert + key).
Start dovecot and then doveadm reload should take long time. Enough for noticing that dovecot stops processing clients.
Aki Tuomi Dovecot oy
That is something that will happen later. Can't give any date, but it's in our internal tasklist.
Ok, thanks.
Just making sure that this (stopping processing clients) and lazy-loading of thousands of SSL certs itself are treated by dovecot team as two separate issues (and tons of SSL certs simply helps to notice first issue).
And was hoping that stopping processing clients issue is easy/easier to solve (but looks like that's not the case).
Aki
-- Arkadiusz Miśkiewicz, arekm / ( maven.pl | pld-linux.org )
Great! Seems to be working fine for my usage and makes my configs 50% smaller (which is gigantic improvement). Will do more testing though.
Thanks!
A little bit offtopic, but what is the point of using imap/pop SNI? All clients want to connect to their own domain or what?
-- Kaspars
On Nov 11, 2016, at 1:29 PM, KSB <listeem@ksb.id.lv> wrote:
Great! Seems to be working fine for my usage and makes my configs 50% smaller (which is gigantic improvement). Will do more testing though.
Thanks!
A little bit offtopic, but what is the point of using imap/pop SNI? All clients want to connect to their own domain or what?
Yes. It allows more comprehensive branding via the domain name.
-FG
On Friday 11 of November 2016, KSB wrote:
Great! Seems to be working fine for my usage and makes my configs 50% smaller (which is gigantic improvement). Will do more testing though.
Thanks!
A little bit offtopic, but what is the point of using imap/pop SNI?
All clients want to connect to their own domain or what?
Yes.
-- Arkadiusz Miśkiewicz, arekm / ( maven.pl | pld-linux.org )
On Nov 11, 2016, at 9:06 AM, Aki Tuomi <aki.tuomi@dovecot.fi> wrote:
If you are interested in testing, please find patch attached that allows you to specify
local_name *.foo.bar { }
or
local_name *.*.foo.bar { }
Dear Aki et al.,
How straightforward would it be to implement the following or similar syntax:local_name foo.tld www.foo.tld mail.foo.tld bar.tld { ... }
??
Thank you!-FG
participants (4)
-
Aki Tuomi
-
Arkadiusz Miśkiewicz
-
Felipe Gasper
-
KSB