Re: [Dovecot] Architecture for large Dovecot cluster
This went to me only so bringing back on list.
On 1/24/2014 11:09 AM, Tom Johnson wrote:
Is anybody using the Object Storage plugin for large-scale installations?
I've not used it.
We're considering it, but are thinking of an in-house S3 storage system (riak, or ceph, or ?) Looking to support perhaps 300k users. I was thinking that if we use a bank of dovecot servers (with director) with ssds as cache, we might be able to consolidate all the storage on something like a riak cluster, which would make scaling simple and inexpensive - certainly much less than a NetApp solution.
Everything costs less than a NetApp...except an EMC.
If anyone has any first-hand experience (or even off-the-top-of-their-head thoughts), I'd love to hear them)
Distributed filesystems give you the advantage of a single filesystem namespace with massive amounts of storage, fairly easy addition of storage space, and distributed replication to allow failure of a storage node without service interruption.
Replication mitigates node failure, but not disk failure, so you still need RAID in each node. So you have RAID6 in a node and filesystem block mirroring amongst nodes. Thus storage utilization is -worse- than direct attach, CFS on SAN, or NFS head attached RAID10 and far worse than RAID6 in these 3 setups. And if using large SSD cache you'd surely use RAID6 with DAS, CFS, or NFS. You'd need half as many disk drives vs DFS.
Each DFS expansion, assuming the typical model, entails the cost of a server, RAID HBA (unless using md) and disks, not strictly buying disks as with DAS, CFS/SAN, or NFS filer. Then you also need more switch ports, more power connections, greater UPS capacity due to all the CPUs, RAM, etc in the nodes. And you'll have a higher electric bill.
So while a distributed filesystem storage architecture may seem less expensive it may not be. And just as one can build a DIY DFS cluster, one can also build a DIY NFS cluster instead of buying a NetApp, saving significant cash on the front end box and on disks since you'd need half as many vs a distributed filesystem architecture, though failure of one node may not be quite as graceful as with a NetApp losing a controller board.
-- Stan
On Jan 24, 2014, at 7:23 PM, Stan Hoeppner <stan@hardwarefreak.com> wrote:
On 1/24/2014 11:09 AM, Tom Johnson wrote: Is anybody using the Object Storage plugin for large-scale installations?
I've not used it.
We're considering it, but are thinking of an in-house S3 storage system (riak, or ceph, or ?) Looking to support perhaps 300k users. I was thinking that if we use a bank of dovecot servers (with director) with ssds as cache, we might be able to consolidate all the storage on something like a riak cluster, which would make scaling simple and inexpensive - certainly much less than a NetApp solution.
Everything costs less than a NetApp...except an EMC.
If anyone has any first-hand experience (or even off-the-top-of-their-head thoughts), I'd love to hear them)
(Stan gives a great run-down on the economics of using a NetApp or even homegrown NFS filer versus using an object storage backend.)
I am quite familiar with NetApp, and EMC - I used to have a number of Celera file servers back in my BigFish/FrontBridge days.
But now I'm in a situation where I have dozens of servers with spare storage bays and unused CPU cycles sitting in data centers where the power is already provisioned, and a DFS is what makes most sense for me now.
So, I would like to ask once again- is anyone on the list using the object storage plugin for dovecot at any reasonably large scale, whether it's an in-house storage solution or S3?
Thanks-
Tom
On 1/26/2014 11:45 AM, Tom Johnson wrote:
On Jan 24, 2014, at 7:23 PM, Stan Hoeppner <stan@hardwarefreak.com> wrote:
On 1/24/2014 11:09 AM, Tom Johnson wrote: Is anybody using the Object Storage plugin for large-scale installations?
I've not used it.
We're considering it, but are thinking of an in-house S3 storage system (riak, or ceph, or ?) Looking to support perhaps 300k users. I was thinking that if we use a bank of dovecot servers (with director) with ssds as cache, we might be able to consolidate all the storage on something like a riak cluster, which would make scaling simple and inexpensive - certainly much less than a NetApp solution.
Everything costs less than a NetApp...except an EMC.
If anyone has any first-hand experience (or even off-the-top-of-their-head thoughts), I'd love to hear them)
(Stan gives a great run-down on the economics of using a NetApp or even homegrown NFS filer versus using an object storage backend.)
Tom I'm sorry I wasted your time with my initial response.
I am quite familiar with NetApp, and EMC - I used to have a number of Celera file servers back in my BigFish/FrontBridge days.
But now I'm in a situation where I have dozens of servers with spare storage bays and unused CPU cycles sitting in data centers where the power is already provisioned, and a DFS is what makes most sense for me now.
Had I known these details above up front I wouldn't have responded. I incorrectly assumed you were designing new infrastructure, wading into new waters, because few are yet to deploy DFS for mailbox storage these days.
So, I would like to ask once again- is anyone on the list using the object storage plugin for dovecot at any reasonably large scale, whether it's an in-house storage solution or S3?
I'm hoping, as I'd guess you are, that someone will respond who is already doing this. If someone has it working well it offers others more storage options, which is always a good thing. Whether it costs more or less than the other solutions, it may still be a better option for some folks either way.
-- Stan
Hi Stan-
(Stan gives a great run-down on the economics of using a NetApp or even homegrown NFS filer versus using an object storage backend.)
Tom I'm sorry I wasted your time with my initial response.
No, you absolutely didn't waste my time, and it was certainly of great advantage to the list. I think it was a great write-up of the advantages and disadvantages of each different option. I know my situation isn't the standard one...
I am quite familiar with NetApp, and EMC - I used to have a number of Celera file servers back in my BigFish/FrontBridge days.
But now I'm in a situation where I have dozens of servers with spare storage bays and unused CPU cycles sitting in data centers where the power is already provisioned, and a DFS is what makes most sense for me now.
Had I known these details above up front I wouldn't have responded. I incorrectly assumed you were designing new infrastructure, wading into new waters, because few are yet to deploy DFS for mailbox storage these days.
I think it's great that you did respond, and thanks for doing so. I know that this is wading into new waters...I'm just hoping I'm not really the very first :)
So, I would like to ask once again- is anyone on the list using the object storage plugin for dovecot at any reasonably large scale, whether it's an in-house storage solution or S3?
I'm hoping, as I'd guess you are, that someone will respond who is already doing this. If someone has it working well it offers others more storage options, which is always a good thing. Whether it costs more or less than the other solutions, it may still be a better option for some folks either way.
Dovecot's commercial arm is certainly marketing the object storage. I'm just hoping someone is actually using it and can offer some guidance.
Tom
On 1/27/2014 11:25 PM, Thomas Johnson wrote:
Hi Stan-
(Stan gives a great run-down on the economics of using a NetApp or even homegrown NFS filer versus using an object storage backend.)
Tom I'm sorry I wasted your time with my initial response.
No, you absolutely didn't waste my time, and it was certainly of great advantage to the list. I think it was a great write-up of the advantages and disadvantages of each different option. I know my situation isn't the standard one...
I am quite familiar with NetApp, and EMC - I used to have a number of Celera file servers back in my BigFish/FrontBridge days.
But now I'm in a situation where I have dozens of servers with spare storage bays and unused CPU cycles sitting in data centers where the power is already provisioned, and a DFS is what makes most sense for me now.
Had I known these details above up front I wouldn't have responded. I incorrectly assumed you were designing new infrastructure, wading into new waters, because few are yet to deploy DFS for mailbox storage these days.
I think it's great that you did respond, and thanks for doing so. I know that this is wading into new waters...I'm just hoping I'm not really the very first :)
So, I would like to ask once again- is anyone on the list using the object storage plugin for dovecot at any reasonably large scale, whether it's an in-house storage solution or S3?
I'm hoping, as I'd guess you are, that someone will respond who is already doing this. If someone has it working well it offers others more storage options, which is always a good thing. Whether it costs more or less than the other solutions, it may still be a better option for some folks either way.
Dovecot's commercial arm is certainly marketing the object storage. I'm just hoping someone is actually using it and can offer some guidance.
Tom
This may be a dumb suggestion and maybe you already have done so, but since this is a commercial only option, maybe you should contact Timo directly and see if he can point you to other customers who have deployed it.
-- Stan
participants (3)
-
Stan Hoeppner
-
Thomas Johnson
-
Tom Johnson