We (a small college with about 3000 active accounts) are currently in the process of moving from UW IMAP running on linux to dovecot running on a cluster of 3 or 4 new faster Linux machines. (Initially using perdition to split the load.)
As we are building and designing the system, I'm attempting to take (or find) benchmarks everywhere I can in order to make informed decisions and so that when we need something faster we have something to compare it to. I've looked all over the Internet and I haven't seen very many good benchmarks running on recent hardware for things like this. Even simple things like how different raid levels perform with fast CPUs and really fast disks.
We recently got the first of the cluster machines (which has the following specs: (from penguin computing) 2x 1.6ghz woodcrest (so 4 cores total) 2g ram (we may upgrade to more, depending on what we seem to need) 6x 15k SAS 73G hard drives.
I've always read that RAID 10 is the RAID of choice for mail, because the checksums take time and whatnot, but on a system that fast I'm sure that's not an issue (which I could see from the cpu usage during benchmarks.) We were also wondering whether RAID6 would be a good option since it would give more redundancy. (raid6 allows 2 drive failures)
All tests were run on ext3 with all 6 disks in the raid. Everything is using Linux software raid. I used bonnie++ to run the benchmarks, and here are the initial results we got: Output Input RAID "Per Chr" "Block" "Per Chr" "Block" RAID6 34628, 97983, 23316, 39077 RAID5 39559, 32715, 32036, 52231 RAID10 38816, 229062, 40645, 267418
I assume that mail running maildir (as we intend to do) is somewhere between writing one char at a time and writing huge blocks to fill 4g files. Since the Per Chr values seemed to have maxed out the CPU and we'll have 4 cores available, I tried the same thing running 4 copies of bonnie at once and adding the results. (I worked backwards and didn't bother with raid 6 for obvious reasons) Output Input RAID "Per Chr" "Block" "Per Chr" "Block" RAID5 37465, 37095, 102828, 174263 RAID10 111239, 128287, 93400, 230320
So, if our system is ever running full out, basically disk bound, it should be able to handle about 4X as many writes and at least as many reads using RAID10 than raid 5, and since raid 6 was pretty similar to raid 5 in the first tests, I assume it will be here too.
I have the full bonnie results if anyone's interested.
I also plan to come up with some basic benchmarks as we start to actually have dovecot working as we want it to. (Possibly using the macs in our labs and XGrid as a "client".. finally a use for that thing :) )
Do people have suggestions as to what to test for a primarily IMAP (+ a little POP) server? I was thinking of doing something similar to http://www.courier-mta.org/mbox-vs-maildir/ which selects a mail box, deletes and expunges a message, fetches a few messages, and then does some searches.
Also, while I have the opportunity, are there any other benchmarks I should run? I suppose I could compare dovecot with mbox vs maildir, with indexing and without. Dovecot with indexing but the indexes deleted, etc. I could compare dovecot to uw imap.
Anything else? I probably can't justify the time to get cyrus up and running. (I did that a few months ago when deciding which server to use...)
Ethan Sommer
--
Ethan Sommer Systems Administrator Gustavus Adolphus College 507-933-7042 sommere@gac.edu
not sure how I messed that up so badly, the first set of numbers were off. here's the first numbers again: Output Input RAID "Per Chr" "Block" "Per Chr" "Block" RAID6 38918, 60365, 40098, 301104 RAID5 38689, 64907, 40320, 282653 RAID10 38816, 229062, 40645, 267418
All numbers are KByes/second. So the raid 6 can do block reads at about 300megabytes per second (for one thread) and the 4 threaded raid10 bellow did 230megabytes per second with 4 threads.
Too many numbers in too big a spreadsheet I guess. In any case, that's what I actually got.
Ethan Sommer wrote:
We (a small college with about 3000 active accounts) are currently in the process of moving from UW IMAP running on linux to dovecot running on a cluster of 3 or 4 new faster Linux machines. (Initially using perdition to split the load.)
As we are building and designing the system, I'm attempting to take (or find) benchmarks everywhere I can in order to make informed decisions and so that when we need something faster we have something to compare it to. I've looked all over the Internet and I haven't seen very many good benchmarks running on recent hardware for things like this. Even simple things like how different raid levels perform with fast CPUs and really fast disks.
We recently got the first of the cluster machines (which has the following specs: (from penguin computing) 2x 1.6ghz woodcrest (so 4 cores total) 2g ram (we may upgrade to more, depending on what we seem to need) 6x 15k SAS 73G hard drives.
I've always read that RAID 10 is the RAID of choice for mail, because the checksums take time and whatnot, but on a system that fast I'm sure that's not an issue (which I could see from the cpu usage during benchmarks.) We were also wondering whether RAID6 would be a good option since it would give more redundancy. (raid6 allows 2 drive failures)
All tests were run on ext3 with all 6 disks in the raid. Everything is using Linux software raid. I used bonnie++ to run the benchmarks, and here are the initial results we got: Output Input RAID "Per Chr" "Block" "Per Chr" "Block" RAID6 34628, 97983, 23316, 39077 RAID5 39559, 32715, 32036, 52231 RAID10 38816, 229062, 40645, 267418
I assume that mail running maildir (as we intend to do) is somewhere between writing one char at a time and writing huge blocks to fill 4g files. Since the Per Chr values seemed to have maxed out the CPU and we'll have 4 cores available, I tried the same thing running 4 copies of bonnie at once and adding the results. (I worked backwards and didn't bother with raid 6 for obvious reasons) Output Input RAID "Per Chr" "Block" "Per Chr" "Block" RAID5 37465, 37095, 102828, 174263 RAID10 111239, 128287, 93400, 230320
So, if our system is ever running full out, basically disk bound, it should be able to handle about 4X as many writes and at least as many reads using RAID10 than raid 5, and since raid 6 was pretty similar to raid 5 in the first tests, I assume it will be here too.
I have the full bonnie results if anyone's interested.
I also plan to come up with some basic benchmarks as we start to actually have dovecot working as we want it to. (Possibly using the macs in our labs and XGrid as a "client".. finally a use for that thing :) )
Do people have suggestions as to what to test for a primarily IMAP (+ a little POP) server? I was thinking of doing something similar to http://www.courier-mta.org/mbox-vs-maildir/ which selects a mail box, deletes and expunges a message, fetches a few messages, and then does some searches.
Also, while I have the opportunity, are there any other benchmarks I should run? I suppose I could compare dovecot with mbox vs maildir, with indexing and without. Dovecot with indexing but the indexes deleted, etc. I could compare dovecot to uw imap.
Anything else? I probably can't justify the time to get cyrus up and running. (I did that a few months ago when deciding which server to use...)
Ethan Sommer
--
Ethan Sommer Systems Administrator Gustavus Adolphus College 507-933-7042 sommere@gac.edu
Hi,
we have over 6000 active accounts (half pop3, half imap), and switching from UW+mbox to dovecot+maildir was such an incredible performance boost that the same old hardware was more than enough. Also from our experience, disk I/O was critical with mbox, but not now (we keep using raid5 without performance penalties). Test it before you switch.
Aaaaaaaaaaagur.
El Martes, 21 de Noviembre de 2006 03:37, Ethan Sommer escribió:
We (a small college with about 3000 active accounts) are currently in the process of moving from UW IMAP running on linux to dovecot running on a cluster of 3 or 4 new faster Linux machines. (Initially using perdition to split the load.)
As we are building and designing the system, I'm attempting to take (or find) benchmarks everywhere I can in order to make informed decisions and so that when we need something faster we have something to compare it to. I've looked all over the Internet and I haven't seen very many good benchmarks running on recent hardware for things like this. Even simple things like how different raid levels perform with fast CPUs and really fast disks.
We recently got the first of the cluster machines (which has the following specs: (from penguin computing) 2x 1.6ghz woodcrest (so 4 cores total) 2g ram (we may upgrade to more, depending on what we seem to need) 6x 15k SAS 73G hard drives.
I've always read that RAID 10 is the RAID of choice for mail, because the checksums take time and whatnot, but on a system that fast I'm sure that's not an issue (which I could see from the cpu usage during benchmarks.) We were also wondering whether RAID6 would be a good option since it would give more redundancy. (raid6 allows 2 drive failures)
All tests were run on ext3 with all 6 disks in the raid. Everything is using Linux software raid. I used bonnie++ to run the benchmarks, and here are the initial results we got: Output Input RAID "Per Chr" "Block" "Per Chr" "Block" RAID6 34628, 97983, 23316, 39077 RAID5 39559, 32715, 32036, 52231 RAID10 38816, 229062, 40645, 267418
I assume that mail running maildir (as we intend to do) is somewhere between writing one char at a time and writing huge blocks to fill 4g files. Since the Per Chr values seemed to have maxed out the CPU and we'll have 4 cores available, I tried the same thing running 4 copies of bonnie at once and adding the results. (I worked backwards and didn't bother with raid 6 for obvious reasons) Output Input RAID "Per Chr" "Block" "Per Chr" "Block" RAID5 37465, 37095, 102828, 174263 RAID10 111239, 128287, 93400, 230320
So, if our system is ever running full out, basically disk bound, it should be able to handle about 4X as many writes and at least as many reads using RAID10 than raid 5, and since raid 6 was pretty similar to raid 5 in the first tests, I assume it will be here too.
I have the full bonnie results if anyone's interested.
I also plan to come up with some basic benchmarks as we start to actually have dovecot working as we want it to. (Possibly using the macs in our labs and XGrid as a "client".. finally a use for that thing :) )
Do people have suggestions as to what to test for a primarily IMAP (+ a little POP) server? I was thinking of doing something similar to http://www.courier-mta.org/mbox-vs-maildir/ which selects a mail box, deletes and expunges a message, fetches a few messages, and then does some searches.
Also, while I have the opportunity, are there any other benchmarks I should run? I suppose I could compare dovecot with mbox vs maildir, with indexing and without. Dovecot with indexing but the indexes deleted, etc. I could compare dovecot to uw imap.
Anything else? I probably can't justify the time to get cyrus up and running. (I did that a few months ago when deciding which server to use...)
Ethan Sommer
-- Trying to make bits uncopyable is like trying to make water not wet. The sooner people accept this, and build business models that take this into account, the sooner people will start making money again. - Bruce Schneier
Ethan Sommer wrote:
[various RAID benchmark results]
Which RAID10 where you using exactly, RAID1+0 or the native Linux MD RAID10 format? (see "man md" or http://en.wikipedia.org/wiki/RAID#Linux_MD_RAID_10 )
Because using the far (or offset) layout will quite dramatically influence the results.
Regards,
Christian Balzer
Christian Balzer Network/Systems Engineer NOC chibi@gol.com Global OnLine Japan/Fusion Network Services http://www.gol.com/
I created 3 mirrors of 2 drives and then striped them.
I will attempt to do the benchmark again by setting the level to 10.
Are you suggesting that you would expect performance to bet better or worse with linux's raid 10 with 2 mirrors and an even number of drives?
Christian Balzer wrote:
Ethan Sommer wrote:
[various RAID benchmark results]
Which RAID10 where you using exactly, RAID1+0 or the native Linux MD RAID10 format? (see "man md" or http://en.wikipedia.org/wiki/RAID#Linux_MD_RAID_10 )
Because using the far (or offset) layout will quite dramatically influence the results.
Regards,
Christian Balzer
--
Ethan Sommer Systems Administrator Gustavus Adolphus College 507-933-7042 sommere@gac.edu
Ethan Sommer wrote:
I created 3 mirrors of 2 drives and then striped them.
I will attempt to do the benchmark again by setting the level to 10.
Are you suggesting that you would expect performance to bet better or worse with linux's raid 10 with 2 mirrors and an even number of drives?
I'm suggesting to expect far better performance, especially with "far" layout (and one less layer of complexity/latency).
Regards,
Christian
Christian Balzer Network/Systems Engineer NOC chibi@gol.com Global OnLine Japan/Fusion Network Services http://www.gol.com/
participants (3)
-
Christian Balzer
-
Ethan Sommer
-
Joseba Torre