Hi p@rick and thanks for the response.

Michael,

• Michael Wessel michael@think-for-yourself.org:

...

I'm currently (re-)planning my email setup and have been doing some research. I have done some searches and read several threads in the areas of my questions here. While there are some that come close I haven't yet been able to get all my questions answered.

I currently run a postfix, dovecot & roundcube setup and have about 2000 active accounts. I have a separate SMTP server for outbound mail and auth is done against a separate LDAP server. In front of the POP/IMAP server I have another SMTP (4 in parallel actually) server that receives and filters inbound mail through a company specific, proprietary filter before the mail hits the POP/IMAP server. LDAP & SMTP servers are ESXi VMs. Do people use 'real' mail clients to connect and IDLE too? Yes, though not sure of the percentage. Most will likely use webmail, some will use POP and some will use IMAP with "real clients". Right now my guess would be about 20% IMAP with Outlook, Thunderbird and such, 10% POP and the rest webmail.

...

So right now both dovecot and roundcube run on the same box which is a Dell PE2950 with dual quad-core Xeon, 16GB RAM and 6 1TB disks in RAID 6, so only local storage using maildir. So far it's been holding up fine, but it's beginning to show signs of overload now. I also expect an increase in users over the next few months up to somewhere between 10 - 20,000 mail boxes. Hence the re-planning.

My first priority in redesigning my setup is reliability. I definitely need something fail-save and as close to always on as possible. Next is performance. And while the budget is of course limited for the moment I'm setting that aside and will worry about that when the time comes.

Now here is my question(s):

In order to support up to 20,000 mailboxes (distributed over several times-zones so they won't all be used at the same time) with a very reliable service with good performance, what do I actually need?

Do I need(ul) SAN or is it just a "would be nice to have"? If yes, why and what would be appropriate for my needs? Or will a setup with a few more servers like the ones I already have, using something like DRBD and distributing services (imap, http, spamd etc) onto different boxes do? Will the server enforce quota? Yes, default quota is 200MB right now, some have larger quotas and a few of those hit several GB.

What will be the average mailbox size? Since the quota is probably going to go up some I'd guess around 400MB on average.

Do people share content e.g. mailings with attachments that go out to all recipients? No, only on a limited basis (like cc'ing maybe 15 or so people but even

On 6/23/2012 3:25 AM, Patrick Ben Koetter wrote: that's rare) There will be somewhat large attachments involved (20-30MB) but that's mostly between individual users and users outside my system.

What might be the maximum number of clients using the server at one time?

Hard to say with the data at hand. I have a caching IMAP proxy for webmail and that has so far recorded 50 as the highest concurrent connections. So adding IMAP users to that and then extrapolating this to 20000 total boxes I'd say 4-500.

Will all users use the same client product e.g. roundcube?

No, they have their choice of any POP3/IMAP client or webmail

What's your backup strategy? What do you use to backup mailboxes?

I was afraid someone was going to ask that question... there isn't one (it hurts just writing that!) The only "backup" currently in place is redundancy on the hardware-side plus limited (i.e. only parts of the mail store) to disk backup. The VMs are easily replaced, but if my maildir goes up in smoke tomorrow then I will probably follow shortly after! So that's definitely part of what I'm working out here. Wanted to nail down the general approach first though before looking at that.

p@rick

...

it is already enormous overshoot in hardware specs. And i do not really catch why you have "4 in parallel" servers. And finally i cannot understand this dividing of servers just to merging it back using VMWare.

because it is a big difference if you have anything in a single machine or splittet in virtual machines - you can move them at runtime to different hosts and if you run out of ressources

ok - for me it is just likes. You have higher change to have the need to move at the first place doing this :)

Quoting Reindl Harald h.reindl@thelounge.net:

Am 23.06.2012 13:09, schrieb Wojciech Puchar:

...

it is already enormous overshoot in hardware specs. And i do not
really catch why you have "4 in parallel" servers. And finally i cannot understand this dividing of servers just to
merging it back using VMWare.

because it is a big difference if you have anything in a single machine or splittet in virtual machines - you can move them at runtime to different hosts and if you run out of ressources for one of them you can buy a phyisclal machine, add it to the cluster and move the virtual machine without any downtime

if you have all on one machine or VM you are not scaleable

Personally I found going from real hardware E51xx servers to E56xx
servers to give double the performance per same speed and amount of
cores for my mail server. Then moving it onto vmware slowed it down
approx 15%.

Overall still a let win, and using vmware for extra HA and easier
maintenance is deferentially worth the slowdown.

...

Buy cheapest but largest SATA drive and use RAID1 (or RAID1+0) setup.

oh no please do not recommend SATA crap with RAID1 and think it is faster than RAID6 - the additional writes doe snot matter if the whole disk-system is much faster and RAID1 has no benefit in performance

OK i would not recommend anything anymore. Normally my advices are for money.

nobody will use SATA disks for high peformance servers in production - really nobody these days!

at least one person. and getting a bit of money helping other increasing performance of their setup. guess who.

On 23/06/12 18:09, Andrzej A. Filip wrote:

On 06/23/2012 01:23 PM, Reindl Harald wrote:

...

Am 23.06.2012 13:09, schrieb Wojciech Puchar:

...

Finally i would recommend to get rid of RAID6. It's terribly slow on writes and writes are common on mail server. depends, it is slower than RAID5, but safer

...

Buy cheapest but largest SATA drive and use RAID1 (or RAID1+0) setup. oh no please do not recommend SATA crap with RAID1 and think it is faster than RAID6 - the additional writes doe snot mat if the whole disk-system is much faster and RAID1 has no benefit in performance

nobody will use SATA disks for high peformance servers in production - really nobody these days! Could you specify/define your idea of "high performance servers" land border? It may reduce the flame war.

Hi,

With dovecot, you can separate indexes and email, and with dbox/mdbox, have ALT storage, so for instance you could keep your indexes in a RAID10 of SSDs, recent email on a RAID10 of 10kRPM/15kRPM SAS drives, and older email can go on a load of 5k/7.2k SATA drives in RAID6, or on a NAS via NFS.

Note: with *dbox your indexes are the only place your mail flags are kept, so don't risk a single drive or even RAID5 for your index store.

This is what I am moving into production from dovecot 1.x on a single RAID6 array (hardware, LSI controller, 6 10k SAS drives in RAID10) which has served very well for a while but is not getting too small for all our mail. Performance has been good for up to 350 users, average mailbox size >4G, about 25-35k incoming mails per day.

Cheers

Alex

-- This message is intended only for the addressee and may contain confidential information. Unless you are that person, you may not disclose its contents or use it in any way and are requested to delete the message along with any attachments and notify us immediately.

"Transact" is operated by Integrated Financial Arrangements plc Domain House, 5-7 Singer Street, London EC2A 4BQ Tel: (020) 7608 4900 Fax: (020) 7608 5300 (Registered office: as above; Registered in England and Wales under number: 3727592) Authorised and regulated by the Financial Services Authority (entered on the FSA Register; number: 190856)

On 23/06/12 19:21, Wojciech Puchar wrote:

...

ALT storage, so for instance you could keep your indexes in a RAID10 of SSDs, recent email on a RAID10 of 10kRPM/15kRPM SAS drives, and older email can go on a load of 5k/7.2k SATA drives in RAID6, or on a NAS via NFS.

far better solution but still about 2-3 times more $/performance than needed, and more complex than needed.

But at least an improvement

I'd respectfully disagree. If you only keep the most recent few weeks of email you could use reasonably priced SSDs for the indexes and perhaps downgrade to SATA for your "hot" store, both of which should be max 10% of your total space with more than a few months of email. My driving factor was to have different spindle sets for each purpose. Who knows, I might have overspent and could have done it with 3 separate SATA arrays.

OTOH what about an SSD caching kit on your server? Supermicro at least do them (well, my UK vendor offers them). Just have a load of big SATA drives and use the kit for caching. The last time I looked a 256GB kit was about UKP 500.

Cheers

Alex

-- This message is intended only for the addressee and may contain confidential information. Unless you are that person, you may not disclose its contents or use it in any way and are requested to delete the message along with any attachments and notify us immediately.

"Transact" is operated by Integrated Financial Arrangements plc Domain House, 5-7 Singer Street, London EC2A 4BQ Tel: (020) 7608 4900 Fax: (020) 7608 5300 (Registered office: as above; Registered in England and Wales under number: 3727592) Authorised and regulated by the Financial Services Authority (entered on the FSA Register; number: 190856)