On Fri, 2010-08-06 at 16:27 -0400, Patrick Domack wrote:

.

Looking at the new methods for v2.0, still on the fence though with them.

Yeah, know what ya mean, I've decided, if it aint broke, dont break it :)

Noel Butler put forth on 8/6/2010 4:29 PM:

Actually you will not notice any difference. How do you think all the big boys do it now :) Granted some opted for the SAN approach over NAS, but for mail, NAS is better way to go IMHO and plenty of large services, ISP, corporations, and universities etc, all use NAS.

The protocol overhead of the NFS stack is such that one way latency is in the 1-50 millisecond range, depending on specific implementations and server load. The one way latency of a fibre channel packet is in the sub 100 microsecond range and is fairly immune to system load. The performance of fibre channel is equal to local disk plus approximately one millisecond of additional effective head seek time due to switch latency, SAN array controller latency, and latency due to cable length. A filesystem block served out of SAN array controller cache returns to the kernel quicker than a block read from local disk that is not in cache because the former suffers no mechanical latency. Due to the complexity of the stack, NFS is far slower than either.

Those who would recommend NFS/NAS over fibre channel SAN have no experience with fibre channel SANs. I'm no fan of iSCSI SANs due to the reliance on TCP/IP for transport, and the low performance due to stck processing. However, using the same ethernet switches for both, iSCSI SAN arrays will also outperform NFS/NAS boxen by a decent margin.

Regarding the OP's case, given the low cost of new hardware, specifically locally attached RAID and the massive size and low cost of modern disks, I'd recommend storing user mail on the new mail host. It's faster and more cost effective than both NFS/SAN. Unless his current backup solution "requires" user mail dirs to be on that NFS server for nightly backup, local disk is definitely the way to go. Four 300GB 15k SAS drives on a good PCIe RAID card w/256-512MB cache in a RAID 10 configuration would yield ~350-400MB/s of real filesystem bandwidth, seek throughput equivalent to a 2 disk stripe--about 600 random seeks/s, 600GB of usable space, ability to sustain two simultaneous disk failures (assuming 1 failure per mirror pair), and cost effectiveness.

-- Stan

On Sat, 2010-08-07 at 09:17 -0600, CJ Keist wrote:

All, Thanks for all the information. I think I'm leaning towards locally attached fiber disk array. Couple of advantages I see, one it will be faster than NFS, second it will allow us to separate user home directory disk quotas and email disk quotas. Something we have been wanting to do for awhile.

You truly wont notice the difference between NAS/SAN in the "real world", we used to use SAN for mail at an old employers, it adds slightly more complexity, and with large volumes of mail, you want things as simple as possible, we found NAS much more reliable, the cost of the units is the same, as netapps do both nas and san, but, if you do not intend to expand beyond the single server (with 3K users you got a long way to go unless you introduce redundancy) then attached fiber disk array will be a cheaper option, even a low end FAS 2k series we use for web was about $30K (to get in this country anyway) obviously much cheaper in the U.S.

What you could do, is talk to vendors, explain what your considering, most often they will send you a loan of devices for a few weeks, so you can configure your scenarios and run the tests, then evaluate which is the best bang for buck way to go, buy be wary of their sales pushes, you know what you want, they don't, they may try upsell you what you'll never ever need.

That said, the one important thing you need to remember, plan for the future.

All the best in your ventures Cheers

On Sat, 2010-08-07 at 18:38 -0400, Jerrale G wrote:

/usr/sbin/dovecot --exec-mail ext /usr/libexec/dovecot/expire-tool

Error: dlopen(/usr/lib64/dovecot/imap/lib11_imap_quota_plugin.so)

expire-tool can't load IMAP plugins. You need the ugly script wrapper shown in http://wiki.dovecot.org/Plugins/Expire

On Sat, Aug 7, 2010 at 3:06 AM, Stan Hoeppner stan@hardwarefreak.comwrote:

Noel Butler put forth on 8/6/2010 4:29 PM:

...

Actually you will not notice any difference. How do you think all the big boys do it now :) Granted some opted for the SAN approach over NAS, but for mail, NAS is better way to go IMHO and plenty of large services, ISP, corporations, and universities etc, all use NAS.

The protocol overhead of the NFS stack is such that one way latency is in the 1-50 millisecond range, depending on specific implementations and server load.

Yes, I would say NFS has greater overhead, but it allows for multi system access where fiber channel does not unless you're using clustered filesystems which have their own issues with latency and lock management.... it's also worth noting that the latencies between the storage and mail processing nodes is an insignificant bottle neck compared to the usual latencies between the client and mail processing nodes.

Those who would recommend NFS/NAS over fibre channel SAN have no experience with fibre channel SANs.

Bold statement there sir :-) From a price performance ratio, I'd argue NAS is far superior and scalable, and generally there is far less management overhead involved with NAS than with SANs, and if you have a commercial high end NAS, you don't have to deal with the idosyncracies of the host file system.

In my previous lives running large scale mail systems handling up to 500k accounts (I work with a team which manages an infrastructure much larger than that now) The price of latency for a single node using NAS flattens out as the number of nodes increase. If you're handling a smaller system with one or two nodes and don't plan or growing significantly, DAS or SAN should be fine.

~Max

Noel Butler put forth on 8/7/2010 5:34 PM:

...

Bold statement there sir :-) From a price performance ratio, I'd argue NAS is far superior and scalable, and generally there is far less management

and with large mail systems, scalability is what it is all about

True large mailbox count scalability requires a "shared nothing" storage architecture and an ultra cheap hardware footprint. The big 3 commercial database vendors all adopted this shared nothing storage strategy a decade ago for scaling OLAP, and then for OLTP. This shared nothing architecture actually works very well for almost any scalable small data transaction application, which includes email.

In a nutshell, you divide the aggregate application data equally across a number of nodes with local storage, and each node is responsible for handling only a specific subset of the total data. I'm guessing this is exactly what Google has done with Gmail, but I've yet to see a white paper detailing the hardware design of gmail, hotmail, or yahoo mail. I'd make a very educated guess that not one of them uses globally shared storage for user mailboxes, like the shared storage we've been discussing.

I would venture to guess that due to performance scalability needs into the tens of millions of mailboxen and, as importantly, geographically distributed scalability, and just as importantly, cost reasons, they probably do something like this mostly shared nothing model.

web server 1 imap server cluster 1 web server 2 ------------------ web server 3 \ / host 1 | 2 disks mirrored | web server 4 \ / ------------------ \ DRDB + GFS ... \ smart / ------------------ / ... director host 2 | 2 disks mirrored | ... / IMAP \ ------------------ web server 509 / proxy \ ------------------ web server 510 / \ host 1 | 2 disks mirrored | web server 511 ------------------ \ DRDB + GFS web server 512 ------------------ / host 2 | 2 disks mirrored | ------------------ imap server cluster 128

An http balancer (not shown) would route requests to any free web server. The smart director behind the web servers contains a database with many metrics and routes new account creation to the proper IMAP server cluster. After the account is established, that can log into any web server but that user's mailbox data transactions are now forever routed to that particular cluster. Each cluster has 1 level of host redundancy and 2 levels of storage redundancy. Each IMAP cluster member would have a relatively low end low power dual core processor, 4GB RAM, 2 x 7.2k RPM disks, and dual GigE ports--a pretty standard base configuration 1U server--and cheap. The target service level being 100-400 concurrent logged in users per IMAP server cluster, for around 50,000 concurrent users for 256 IMAP servers.

This is not a truly shared nothing architecture, as we have an IMAP service based on a 2 node cluster. However, given the total size of these organizations' user bases, in the multiple 10s of millions of mailboxen, in practical terms, this is a shared nothing design, as only a a few dozen to a hundred user mailboxen exist on each server. One host in each cluster pair resides in a different physical datacenter close to the user do a catastrophic network or facility failure doesn't prevent the user from accessing his/her mailbox.

Depending on how much redundancy, and thus money, the provider wishes to pony up, each two node cluster above could be expanded to a node count sufficient to put one member of each cluster in each and every datacenter the provider has. The upside to this is massive redundancy and an enhanced user experience when an outage at one center occurs, or a backbone segment goes down. The downside is data synchronization across WAN links, with an n+1 increase in synchronization overhead for each cluster member added.

Having central shared mailbox storage for this size user count is impossible due to the geographically distributed datacenters these outfits operate. The shared nothing 2 node cluster approach I've suggested is probably pretty close to what these guys are using. If a mailbox server goes down, its cluster partner carries the load for both until the failed node is repaired/replaced. If both nodes go down, a very limited subset of the user base is affected.

If one centralized FC SAN or NFS/NAS array was used per datacenter in place of the local disks in these cheap clusters, costs would go through the roof. To duplicate the performance of the 256 x 7.2k local SATA disks (512 total but mirrors don't add to performance), you'd need an array controller with big cache (8-32GB), 40k random IO/s at the spindle level and 7.6GB/s of random IO spindle throughput. This would require an array controller with a minimum of 10 x 8Gb FC ports, or 8 x 10GbE NAS ports, and 128 x 15k SAS disks. Depending on whose unit meeting these specs that you buy, you're looking at somewhere in the neighborhood of $250-500k.

And given the cost of the switch and HBA infrastructure required in this central storage scenario, those 256 single socket cheap IMAP cluster machines are going to rapidly turn into 8 rather expensive dual socket 12 core processor nodes (24 cores per node, 192 total cores) with 128GB RAM each, 1TB total, same as the 256 el cheapo node aggregate. Each node will have an 8Gb FC HBA or 10GbE HBA, and a single connection to the SAN/NAS array controller, eliminating the need/cost for a dedicated switch. As configured, each of these servers will run ~$20k USD due to the 128GB of RAM, the ~$1,000 HBA, and due to the fact that vendors selling such boxen gouge customers on big memory configurations. Base price for the box with 2 x 12 core Opteron and 16GB RAM is ~$6k USD. Anyway, figure 8 x $20k = ~$160,000 for the IMAP cluster nodes. Add in $250-$500k for the SAN/NAS array, and you're looking at ~$410k to ~$660k.

A quantity buy of 256 of the aforementioned cheap single socket boxen will get the price down to well less than $1,000 each, probably more like $800, yielding a total cluster cost of about $200k USD for 256 cluster hosts--less than half that of the big smp SAN/NAS solution.

The cluster host numbers I'm using are merely examples. Google for example probably has a larger IMAP cluster server count per datacenter than the 256 nodes in my example--that's only about 6 racks packed with 42 x 1U servers. Given the number of gmail accounts in the US, and the fact they have less than 2 dozen datacenters here, we're probably looking at thousands of 1U IMAP servers per datacenter.

-- Stan

Hi Stan,

On Sat, Aug 7, 2010 at 10:27 PM, Stan Hoeppner stan@hardwarefreak.comwrote:

Care to elaborate on this point? The NFS server sits in user space. All cluster filesystem operations take place in kernel space.

If you expand your definition of NFS server to include high end systems (NetApp being a common example), the placement of the NFS server isn't necessarily limited to user space. Some vendors like BlueArc, use FPGAs to handle the protocols. ORT in many cases is less than 1 msec on some of these boxes.

econd range.

Dovecot clusters may be simpler to implement using NFS storage servers,

Simpler and more cost effective. The price / performance (per watt if you want to go that far seeing as you don't need 2 fabrics ) generally favor NAS or some other kind of distributed file system based approach. The gains that come from parallelization are worth it at the cost of slightly less performance on an individual node basis, especially if you're dealing with N+2 or greater availabilty schemes.

These Distributed File Systems and specialilzed RPC mechanisms have higher overhead than even NFS, but they make up for it by increasing paralleization and using some very creative optimizations that you can use when you have many multiple machines and some other things that are don't have useful analogs outside of Google.

|In a nutshell, you divide the aggregate application data equally across a |number of nodes with local storage, and each node is responsible for handling |only a specific subset of the total data.

You can go the same thing with NFS nodes, with the added benefit using the automounter (on the low end) to "virtualize" the name space similar to what they do with render farms.

|The cluster host numbers I'm using are merely examples. Google for example |probably has a larger IMAP cluster server count per datacenter than the 256 |nodes in my example--that's only about 6 racks packed with 42 x 1U servers. |Given the number of gmail accounts in the US, and the fact they have less than |2 dozen datacenters here, we're probably looking at thousands of 1U IMAP |servers per datacenter.

The architecture you describe is very similar to webex, but they cap the number of accounts per node at some ridiculously small level, like 10,000 or something and use SAS drives.

~Max

The NFS server sits in user space. Oops? I don't know what Linux does, but with BSD, it has always been in-kernel.

On 8/9/10 5:31 AM, Bernd Petrovitsch wrote:

On Mon, 2010-08-09 at 12:18 +0200, Edgar Fuß wrote:

...

...

The NFS server sits in user space. Oops? I don't know what Linux does, but with BSD, it has always been in-kernel.

Historically there was a user-space NFS-daemon (and can very probably be found via Google today). Actually there are stories about people using it because if you export filesystems via a user-space NFS daemon, you can change the mounting below without affecting the clients for NFS-v3.

But the kernel has a NFS-server since years and all (somewhat common) distributions use it per default.

Debian still gives you a choice with packages nfs-user-server and nfs-kernel-server.

~Seth

On Sun, 2010-08-08 at 00:27 -0500, Stan Hoeppner wrote:

Care to elaborate on this point? The NFS server sits in user space. All cluster filesystem operations take place in kernel space.

hellloooooooo 2001 is calling you, NFS has been in kernel for many years now.

it seems I'm missing a lot of your posts, looks likely our SA anti_troll rules

On Tue, 2010-08-10 at 06:21 -0400, Charles Marcus wrote:

Simply being wrong does not a troll make, however...

No , what makes one a troll is the amount of constant crap emitted from a person over many mediums.

It was pointed out to you a couple of months ago on the spam-l list that

spam-l? they are IMHO mostly just chest beating wankers trying to justify their own self importance. The only person their I actually have heard of before in my 20 odd years of doing this, and respect, is Al, and he rarely posts.

you are making some really bad decisions wrt your SA/Mailscanner rules,

in your opinion, and yours alone

some of which make you worse than a backscatter source - in fact, one of your rules caused you to actually spam the spam-l list, and then to

stop spreading lies. it wasnt spam at all as you know, the content of the message was direct and legit, identical to many others posts, the only difference is I do not whitelist outbound mail, so it was caught, and issued a warning about the included content, ill never change that and make no apologies, ever. i dont care that you disagree with that.

compound the error you explicitly said you made no apologies for it (which got you moderated) - not very smart...

Correct and it goes to show what childish morons exist in such position, I'll gladly put up on the web a copy of the archives for anyone (their copyright BS is invalid and legally unenforceable in this country because it is a "general mailing list that anyone is free to join"), and if you only saw the immature private mail terranson sent me :)

You also said earlier that you were missing some of my posts - and now some of Stans - which makes it on the order of 100% certain that you are losing other legitimate mail, so obviously that rule is not the only ill-advised one you implement.

negative, we have rules in place for ranters and trolls, we used to run an nntp-email gateway, so the rules were perfected over time, the rules ARE working EXACTLY as designed when they catch Stans 10 page rants, but when he behaves himself, like the 5 liner he did a week ago, they have a chance of getting through.

In relation to you, the email service you use has a poor reputation here, that along with the troll rules will certainly ensure I dont see your posts unless I am so bored I go looking in archives (only went looking for this after someone sent me a weird email a few days back and I was curious what he was going on about) and in accordance with that, I luckily wont see any self justification rant you try reply with, so dont bother, when a netblock or host is listed for miscreant activities, it tends to mean they are out forever, this includes high score additions in SA as well.

To each his own, just please don't start spamming the dovecot list.

oh grow up little child, I have been on this list for many many many years, so keep your high and mighty self importance to the spam-l (does L stand for "lamers"... starting to think so) list.

my apologies to anyone else who was bored enough to read this, but experience shows sometimes you have to lower yourself to a childs level to communicate with that child. (and I bet he still replies to me cause he still just wont get that fact that I wont see it *sigh*)