[Dovecot] Dovecot and SATA Backend
Hi all,
I plan to run a dovecot IMAPS and POPS service on our network. We handle about 3 000 mailboxes. I thought first buying a topnotch server (8 cores and 16 Go RAM) with equalogic iSCSI SAN SAS 15K for storage backend.
On second though (and after a comprhensive read of dovecot features), I saw in http://wiki.dovecot.org/MailLocation that index files can be created on a separete local filesystem. My question is : for 3000 users, is it possible to have only a SATA backend attached to my topnotch server (to handle bigger mail quotas) by storing index on local hard drives (SAS drives) ?
Extra question, what is the better : iSCSI SATA backend or NFS share ? NFS share is more convenient to have a failover server.
Thanks for your help.
Regards,
-- Nicolas Grenèche - Orléans University - France http://blog.garnett.fr (in french)
Quoting Nicolas GRENECHE nicolas.greneche@gmail.com:
I plan to run a dovecot IMAPS and POPS service on our network. We handle about 3 000 mailboxes. I thought first buying a topnotch server (8 cores and 16 Go RAM) with equalogic iSCSI SAN SAS 15K for storage backend.
Sounds like overkill to me, but if you have the money go for it. :)
I run mine on an 8 core (dual quad core) system with 4G RAM, using SATA (would have preferred SAS, but cost was an issue for us).
created on a separete local filesystem. My question is : for 3000 users, is it possible to have only a SATA backend attached to my topnotch server (to handle bigger mail quotas) by storing index on local hard drives (SAS drives) ?
Sure. If you only have one dovecot server without any failover this is fine. If you have multiple (active or passive) servers then more care is required, and you need to decide on the level of risk you want to take.
Extra question, what is the better : iSCSI SATA backend or NFS share ?
iSCSI would be better than NFS IMHO.
NFS share is more convenient to have a failover server.
If you introduce NFS and/or a failover server, your local index question gets much more complex...
Is that a design requirement, desire, or future option?
Thanks for your help.
Regards,
-- Nicolas Grenèche - Orléans University - France http://blog.garnett.fr (in french)
-- Eric Rostetter The Department of Physics The University of Texas at Austin
This message is provided "AS IS" without warranty of any kind, either expressed or implied. Use this message at your own risk.
2009/11/16 Eric Jon Rostetter eric.rostetter@physics.utexas.edu:
Quoting Nicolas GRENECHE nicolas.greneche@gmail.com:
I plan to run a dovecot IMAPS and POPS service on our network. We handle about 3 000 mailboxes. I thought first buying a topnotch server (8 cores and 16 Go RAM) with equalogic iSCSI SAN SAS 15K for storage backend.
Sounds like overkill to me, but if you have the money go for it. :)
I prefer disk space to useless storage velocity ;-)
I run mine on an 8 core (dual quad core) system with 4G RAM, using SATA (would have preferred SAS, but cost was an issue for us).
created on a separete local filesystem. My question is : for 3000 users, is it possible to have only a SATA backend attached to my topnotch server (to handle bigger mail quotas) by storing index on local hard drives (SAS drives) ?
Sure. If you only have one dovecot server without any failover this is fine. If you have multiple (active or passive) servers then more care is required, and you need to decide on the level of risk you want to take.
Extra question, what is the better : iSCSI SATA backend or NFS share ?
iSCSI would be better than NFS IMHO.
NFS share is more convenient to have a failover server.
If you introduce NFS and/or a failover server, your local index question gets much more complex...
I know it, index may be handled on NFS server on fast local hardrives. The whole stuff should be exported to dovecot server.
Is that a design requirement, desire, or future option?
It should be a future option, but index management will be more tricky as you stated.
Thanks for your help.
Regards,
-- Nicolas Grenèche - Orléans University - France http://blog.garnett.fr (in french)
-- Eric Rostetter The Department of Physics The University of Texas at Austin
This message is provided "AS IS" without warranty of any kind, either expressed or implied. Use this message at your own risk.
Quoting Nicolas GRENECHE nicolas.greneche@gmail.com:
It should be a future option, but index management will be more tricky as you stated.
If you want to do any kind of clustering/failover, even in the future, then I would go with iSCSI/SAN of some sort instead of NFS... Just my $0.02.
The other way to think about it is that if it is a future plan then it won't matter much as Timo will probably get NFS support working so well by then it won't matter any more. :) But, right now, NFS support is a bit tricky, though constantly improving, and I'd still recommend you stay away from it if possible...
-- Eric Rostetter The Department of Physics The University of Texas at Austin
This message is provided "AS IS" without warranty of any kind, either expressed or implied. Use this message at your own risk.
El Lunes 16 Noviembre 2009 a las 14:00, Nicolas GRENECHE escribió:
Extra question, what is the better : iSCSI SATA backend or NFS share ? NFS share is more convenient to have a failover server.
In my experience, NFS for maildirs is fast enough, and well known and supported. For indeces, in my benchmarks iSCSI is 3 times faster than nfs, and almost as fast as local disk. But I guess this depends a lot on your exact setup.
-- Joseba Torre. Vicegerencia de TICs, área de Explotación
Extra question, what is the better : iSCSI SATA backend or NFS share ? NFS share is more convenient to have a failover server.
Everyone wants to use nfs for mailstorage, because it is convenient, but nfs is the wrong storage model for mail. NFS shines on big files but metadata performance is low, too much overhead!
We currently have a mailcluster with 10000 mailboxes and the data ist stored on a netapp storage. If you have a Mailbox with more then 1000 mails, it gets slow over nfs.
Our new architecuture will look like this:
No NFS, no iSCSI we use DAS. There will be multiple dedicated mailbox server in pairs with a small postfix instance and dovecot. These Servers will only be used for delivering/IMAP/POP3. To get HA we use DRBD und Heartbeat in active/passive setup. In the front there will be a IMAP/POP3 proxy to direct the user to his mailbox server.
This setup scales and is cheap, you only have to find a way to distribute the mailboxes around all mailbox servers.
I think rackspace email uses a similar setup with drbd and dovecot.
alex handle schrieb:
Extra question, what is the better : iSCSI SATA backend or NFS share ? NFS share is more convenient to have a failover server.
Everyone wants to use nfs for mailstorage, because it is convenient, but nfs is the wrong storage model for mail. NFS shines on big files but metadata performance is low, too much overhead!
We currently have a mailcluster with 10000 mailboxes and the data ist stored on a netapp storage. If you have a Mailbox with more then 1000 mails, it gets slow over nfs.
Our new architecuture will look like this:
No NFS, no iSCSI we use DAS. There will be multiple dedicated mailbox server in pairs with a small postfix instance and dovecot. These Servers will only be used for delivering/IMAP/POP3. To get HA we use DRBD und Heartbeat in active/passive setup. In the front there will be a IMAP/POP3 proxy to direct the user to his mailbox server.
This setup scales and is cheap, you only have to find a way to distribute the mailboxes around all mailbox servers.
I think rackspace email uses a similar setup with drbd and dovecot.
sorry for the stupid question what is "DAS" do you have a link etc for it, to get more info
-- Best Regards
MfG Robert Schetterer
Germany/Munich/Bavaria
Robert Schetterer robert@schetterer.org writes:
sorry for the stupid question what is "DAS" do you have a link etc for it, to get more info
See http://en.wikipedia.org/wiki/Direct-attached_storage
-- Nicolas
On Fri, Nov 20, 2009 at 12:28 PM, Nicolas KOWALSKI niko@petole.demisel.net wrote:
Robert Schetterer robert@schetterer.org writes:
sorry for the stupid question what is "DAS" do you have a link etc for it, to get more info
to put it simply: local storage
We use Dell R710 in pair, each with 6 15K SAS Disks and RAID 10 - iSCSI or NFS can hardly be faster :)
El Viernes 20 Noviembre 2009 a las 13:05, alex handle escribió:
to put it simply: local storage
We use Dell R710 in pair, each with 6 15K SAS Disks and RAID 10 - iSCSI or NFS can hardly be faster
But "faster" is not always the way to measure things.
In this topic, I think that raw disk access speed is not a critical value; I've tested our setup with maildir on NFS and performance is close enough to local disks. Given this, I prefer the extra features our NAS/SAN setup gives us, like having data replication between phisically separated disks in a couple of clicks ,redundant paths to data, easier management, ...
-- Joseba Torre. Vicegerencia de TICs, área de Explotación
Joseba Torre wrote:
El Viernes 20 Noviembre 2009 a las 13:05, alex handle escribió:
to put it simply: local storage
We use Dell R710 in pair, each with 6 15K SAS Disks and RAID 10 - iSCSI or NFS can hardly be faster
But "faster" is not always the way to measure things.
In this topic, I think that raw disk access speed is not a critical value; I've tested our setup with maildir on NFS and performance is close enough to local disks. Given this, I prefer the extra features our NAS/SAN setup gives us, like having data replication between phisically separated disks in a couple of clicks ,redundant paths to data, easier management, ...
Joseba,
Could you share how you have configured Dovecot in order to achieve replication, redundant path .....?
Regards,
Mario Antonio
El Viernes 20 Noviembre 2009 a las 14:47, Mario Antonio escribió:
Could you share how you have configured Dovecot in order to achieve replication, redundant path .....?
I have configured nothing on dovecot's side. We have a lot of EMC gear providing most of the features, maildirs in a replicated NFS share, and two dovecot boxes beyond a CISCO load balancer (active/passive).
If everything goes as expected, next month we'll move the indexes from local disks to a soft raid of two iSCSI disks; but, again, there's almost nothing on dovecot's side.
HTH
Joseba Torre. Vicegerencia de TICs, área de Explotación
But "faster" is not always the way to measure things.
In this topic, I think that raw disk access speed is not a critical value; I've tested our setup with maildir on NFS and performance is close enough to local disks. Given this, I prefer the extra features our NAS/SAN setup gives us, like having data replication between phisically separated disks in a couple of clicks ,redundant paths to data, easier management, ...
In my test local storage was always ~ 2 times faster than nfs.
if you have the deep pockets to buy all the storage gear and replication licenses/maintainence then it's for sure far better. But if you are an isp, you have to sell the storagespace for pennies. So we came up with a cheap robust and scalable storage solution special to our mailsetup. It Supports HA with blocklevel-replication and snapshots with lvm so it's even some what manageable.
Another point against nfs is that i wanted to avoid potential locking/coruption problems. I know it can be solved by always redirecting the same user to the same server, but then again i loose the convenience of nfs. So i ditch nfs complety and use multiple storage servers in pairs running dovecot.
On 11/16/2009 8:00 AM, Nicolas GRENECHE wrote:
Hi all,
I plan to run a dovecot IMAPS and POPS service on our network. We handle about 3 000 mailboxes. I thought first buying a topnotch server (8 cores and 16 Go RAM) with equalogic iSCSI SAN SAS 15K for storage backend.
We run about 300 mailboxes, ~1 to 1.5 million inbound connections per month and about 1-1.5 million messages delivered to Dovecot per month on a low-end server. The O/S drive is 10k RPM SATA and the MailDir folders are stored on a 4-disk RAID-10 7200rpm SATA. It's a dual-core, ~2GHz, 64bit CentOS 5 server with only 4GB RAM. We also do anti-virus and spam filtering (SpamAssassin) scoring on that system along with using it for a few other tasks. The system chugs at times (during the daily backup window) but otherwise I'd say we're at about 25-30% load currently. Fortunately, providing basic mail service isn't that system intensive.
I think your sizing issue is going to be more about how many messages per month you're pushing through the system then sheer number of mailboxes. And maybe the overall size of the mail store. (Heck, our old mail server was a Solaris 200MHz x86 box with 256MB RAM and a pair of 80GB IDE drives setup before I took over administration of the mail system.)
Our current server was a test case that we put in about 18 months ago. Hopefully next year we can upgrade to better equipment (more cores, more memory, and more and faster spindles). And maybe some HA stuff like DRBD and Heartbeat.
participants (8)
-
alex handle
-
Eric Jon Rostetter
-
Joseba Torre
-
Mario Antonio
-
Nicolas GRENECHE
-
Nicolas KOWALSKI
-
Robert Schetterer
-
Thomas Harold