Scalability with high density servers and proxies, TCP port limits

[Christian Balzer]

Hello,

first post in 3 years, kinda shows how painless Dovecot is. ^o^

Also this isn't really a dovecot issue, alas it's involved and since there
are some large scale implementations of it I hope somebody here has some
insights I might have missed.

Currently we're running this setup:

1. LVS (DR mode) in a HA configuration (2 node cluster)
2. Dovecot in proxy mode on a 2 node cluster
3. Dovecot on actual mailbox servers (dual node DRBD clusters)

There are about 500k users, but most of them use POP3, so there are
usually less than 6k IMAP sesions at any given time.

This is about to change, I'm looking at potentially millions of users who
will have all semi-permanent IMAP sessions.

We already have a pure SSD based mailbox cluster and based on the
experiences with that another one is on order that will be able to easily
handle about 500k users with regards to IOPS and other needs.

However there's the issue of having all these concurrent IMAP sessions.
Namely, running out of ephemeral ports.

Lets assume 2 million users and 50k ports per IP and revisit the setup
above.

1. LVS should have no problem, from experience and tests I expect a well
tuned and spec'ed machine to handle millions of connections.
This is in DR mode, in NAT mode I assume things would run into a wall a
lot quicker.
But even if LVS should run out of steam, there's a wide selection of high
capacity load balancers available.

2. Here is where the fun starts. 
Each IMAP session that gets proxied to the real mailbox server needs a
port for the outgoing connection. 
So to support 2 million sessions we need 40 IP addresses here. Ouch.
And from a brief test having multiple IP addresses per server won't help
either (Dovecot unsurprisingly picks the main IP when establishing a
proxy session to the real mailbox), at least not with just one default GW. 

3. All of this gets repeated on the actual mailbox servers, by either
having a lot of low density servers or (preferably) high density servers
with multiple IP addresses. 

Am I on track so far or missing something obvious?

How many concurrent connections do you (hello Timo) think dovecot in proxy
mode can handle? High performance mode of course in this case.
I'm interested in internal limitations, assume that CPU and RAM are
amply supplied.

Any and all feedback is appreciated.

Regards,

Christian
-- 
Christian Balzer        Network/Systems Engineer                
chibi at gol.com   	Global OnLine Japan/Fusion Communications
http://www.gol.com/