Scalability with high density servers and proxies, TCP port limits
Hello,
first post in 3 years, kinda shows how painless Dovecot is. ^o^
Also this isn't really a dovecot issue, alas it's involved and since there are some large scale implementations of it I hope somebody here has some insights I might have missed.
Currently we're running this setup:
- LVS (DR mode) in a HA configuration (2 node cluster)
- Dovecot in proxy mode on a 2 node cluster
- Dovecot on actual mailbox servers (dual node DRBD clusters)
There are about 500k users, but most of them use POP3, so there are usually less than 6k IMAP sesions at any given time.
This is about to change, I'm looking at potentially millions of users who will have all semi-permanent IMAP sessions.
We already have a pure SSD based mailbox cluster and based on the experiences with that another one is on order that will be able to easily handle about 500k users with regards to IOPS and other needs.
However there's the issue of having all these concurrent IMAP sessions. Namely, running out of ephemeral ports.
Lets assume 2 million users and 50k ports per IP and revisit the setup above.
LVS should have no problem, from experience and tests I expect a well tuned and spec'ed machine to handle millions of connections. This is in DR mode, in NAT mode I assume things would run into a wall a lot quicker. But even if LVS should run out of steam, there's a wide selection of high capacity load balancers available.
Here is where the fun starts. Each IMAP session that gets proxied to the real mailbox server needs a port for the outgoing connection. So to support 2 million sessions we need 40 IP addresses here. Ouch. And from a brief test having multiple IP addresses per server won't help either (Dovecot unsurprisingly picks the main IP when establishing a proxy session to the real mailbox), at least not with just one default GW.
All of this gets repeated on the actual mailbox servers, by either having a lot of low density servers or (preferably) high density servers with multiple IP addresses.
Am I on track so far or missing something obvious?
How many concurrent connections do you (hello Timo) think dovecot in proxy mode can handle? High performance mode of course in this case. I'm interested in internal limitations, assume that CPU and RAM are amply supplied.
Any and all feedback is appreciated.
Regards,
Christian
Christian Balzer Network/Systems Engineer
chibi@gol.com Global OnLine Japan/Fusion Communications
http://www.gol.com/
Hi,
Am 03.07.2015 um 05:14 schrieb Christian Balzer:
- Here is where the fun starts. Each IMAP session that gets proxied to the real mailbox server needs a port for the outgoing connection. So to support 2 million sessions we need 40 IP addresses here. Ouch. And from a brief test having multiple IP addresses per server won't help either (Dovecot unsurprisingly picks the main IP when establishing a proxy session to the real mailbox), at least not with just one default GW.
If I remeber correctly there is a config option in dovecot 2.x where you can set the ip addresses which dovecot should use for outgoing proxy connections. Sorry, but I can't remeber the option.
Best Urban
On Fri, 03 Jul 2015 07:05:43 +0200 Urban Loesch wrote:
Hi,
Am 03.07.2015 um 05:14 schrieb Christian Balzer:
- Here is where the fun starts. Each IMAP session that gets proxied to the real mailbox server needs a port for the outgoing connection. So to support 2 million sessions we need 40 IP addresses here. Ouch. And from a brief test having multiple IP addresses per server won't help either (Dovecot unsurprisingly picks the main IP when establishing a proxy session to the real mailbox), at least not with just one default GW.
If I remeber correctly there is a config option in dovecot 2.x where you can set the ip addresses which dovecot should use for outgoing proxy connections. Sorry, but I can't remeber the option.
Looking at the documentation on the Wiki I was going to say "That won't help", as it says address. http://wiki2.dovecot.org/PasswordDatabase/ExtraFields/Proxy
But since that page is rather terse, I looked up the changelog and found that it indeed was added for use cases like mine: http://www.dovecot.org/list/dovecot-cvs/2014-June/024574.html
Unfortunately the latest dovecot version in Debian is 2.2.13...
Additionally this still leaves the actual mailbox servers, which in my case will need to be able to handle more than 50k sessions as well.
Thanks for the info,
Christian
Christian Balzer Network/Systems Engineer
chibi@gol.com Global OnLine Japan/Fusion Communications
http://www.gol.com/
On Fri, 3 Jul 2015 14:29:55 +0900 Christian Balzer wrote:
On Fri, 03 Jul 2015 07:05:43 +0200 Urban Loesch wrote:
Hi,
Am 03.07.2015 um 05:14 schrieb Christian Balzer:
- Here is where the fun starts. Each IMAP session that gets proxied to the real mailbox server needs a port for the outgoing connection. So to support 2 million sessions we need 40 IP addresses here. Ouch. And from a brief test having multiple IP addresses per server won't help either (Dovecot unsurprisingly picks the main IP when establishing a proxy session to the real mailbox), at least not with just one default GW.
To follow up on myself, with multiple IPs and appropriate(*) iproute rules this works as well.
(*) for each IP in "interfaces" add something like this:
up ip route add 192.168.1.0/24 dev eth0 src 192.168.1.109 table T2
up ip route add default via 192.168.1.1 table T2
up ip rule add from 192.168.1.109 table T2And the tables in "/etc/iproute/rt_tables".
Christian
If I remeber correctly there is a config option in dovecot 2.x where you can set the ip addresses which dovecot should use for outgoing proxy connections. Sorry, but I can't remeber the option.
Looking at the documentation on the Wiki I was going to say "That won't help", as it says address. http://wiki2.dovecot.org/PasswordDatabase/ExtraFields/Proxy
But since that page is rather terse, I looked up the changelog and found that it indeed was added for use cases like mine: http://www.dovecot.org/list/dovecot-cvs/2014-June/024574.html
Unfortunately the latest dovecot version in Debian is 2.2.13...
Additionally this still leaves the actual mailbox servers, which in my case will need to be able to handle more than 50k sessions as well.
Thanks for the info,
Christian
--
Christian Balzer Network/Systems Engineer
chibi@gol.com Global OnLine Japan/Fusion Communications
http://www.gol.com/
participants (2)
-
Christian Balzer
-
Urban Loesch