[Dovecot] Multiple locations, 2 servers - planning questions...

[Stan Hoeppner]

On 3/1/2012 5:43 AM, Charles Marcus wrote:
> On 2012-02-29 9:15 PM, Stan Hoeppner <stan at hardwarefreak.com> wrote:
>> Q:  How many concurrent IMAP clients could you serve with this setup
>>      before hitting a bottleneck at any point in the architecture?
> 
> No idea how to calculate it...

The correct answer is approximately 120,000 concurrent users, based on
an assumed average of ~3MB-5MB of ram consumed in all processes for each
user.

>> What is the first bottleneck you'd run into?
> 
> Unless this is a trick question, the OC-12 link (since it is only
> 644Mb), and the next bottleneck would be the 2 GbE server connections to
> the router (are these bonded? if so, what mode?...

It was a bit of a trick question, with a somewhat elaborate setup,
designed to shift your focus/thinking.  Apparently I failed in my effort
here.  The correct answer is that RAM will be the first bottleneck.
Then disk IOPS, finally followed by the OC-12 assuming we beef up the
others.

> Since the vast majority of our connections will be *local*, I'm
> unconcerned about the internet connect speeds (one office has a 100/10Mb
> Cable (Comcast Business Class) connection, the other will have a
> 100/100Mb fiber/ethernet connection).

You didn't grasp why I used the OC-12 in my example.  It had nothing to
do with LAN/WAN, local or remote, but the total users/traffic a 600Mb/s
link can carry.

> My main priority is that the user experience at each physical location
> be optimal, which is why I'm more focused on making sure each offices
> users are connected to only the local server for all services
> (file/print/mail).

A single MAN (Metropolitan Area Network) 1000BASE-LX link, good for 5km,
likely what you will have, is more than sufficient to carry the 2nd
office site traffic while keeping all of your servers/etc where they are
now.

> My choices are, as I see it, single GbE connections, or add some
> multiport GbE cards (these Dells support up to 3 PCIe cards) and bond
> some ports together for each VM. 10GbE is simply not in our price range
> (and I don't think we need it anyway), although I did stumble on these
> while googling and am waiting on pricing, since they claim to be 'much
> cheaper':

With specs like that you must be supporting 100,000 users. ;)

> Obviously, I don't have the experience or expertise to answer these
> questions myself (never analyzed IMAP traffic to have an idea of the
> bandwidth each user uses, and probably wouldn't trust my efforts if I
> made the attempt). Hopefully, there are some people here who have a
> rough idea, which is why I brought this question up here.

Your company/employer has less than 250 users IIRC.  Is this right?
You're a media company that works with files much larger than the
average company.  Is that correct?  Let's cut to the chase shall we?

Your 1000BASE-LX MAN link has an after link overhead bandwidth of just
over 100MB/s full duplex.  To put this into real world perspective, you
can copy a single 4.7GB DVD in 47 seconds, or 1 in each direction in the
same time, 2 total, 9.4GB total.  You can copy 20 full DVDs over this
link, 10 in each direction, in less than 8 minutes.  Add heavy IMAP
traffic for 500 concurrent users and it's still less than 10 minutes and
the IMAP users won't have a clue if the switch VLAN QOS is setup correctly.

You see GbE as mundane, slow, because it has been ubiquitous for some
time, being a freebie on both servers and desktops.  This is why I used
the OC-12 example at $15K/month, hoping you'd start to grasp that cost
has little direct relationship to performance.  GbE is "free" now
because the cost of the silicon to drive a 1000MHz signal over 300
meters of copper wire is no longer higher than for 100BASE-T.

Here's another comparison.  All internet backbone links are OC-48 at
2.5Gb/s.  It takes only 2.5 GbE links to equal a backbone link.
Backbone links carry the traffic of *millions* of users, all
applications, all data stream types.  And that's *only* 250MB/s.

So, the point is, a single 1000BASE-LX MAN link is far more than plenty
to carry all of the traffic you'll throw at it, and quite a bit more,
with some minor QOS configuration.  Consider how much money, time, and
duplication of services and servers you are going to save now that you
realize you need nothing other than the 1000BASE-LX MAN link, and closet
switches at the second office site?

Get yourself a qualified network architect.  Pay for a full network
traffic analysis.  He'll attach sniffers at multiple points in your
network to gather traffic/error/etc data.  Then you'll discuss the new
office, which employees/types with move there, and you'll be able to
know almost precisely the average and peak bandwidth needs over the MAN
link.  He'll very likely tell you the same thing I have, that a single
gigabit MAN link is plenty.  If you hire him to do the work, he'll
program the proper QOS setup to match the traffic patterns gleaned from
the sniffers.

-- 
Stan