[Dovecot] OT - small hd recommendation
A little OT - but I've seen a few opinions voiced here by various admins and I'd like to benefit.
Currently running a single combined server for multiple operations - fileserver, mailserver, webserver, virtual server, and whatever else pops up. Current incarnation of the machine, after the last rebuild, is an AMD Opteron 4180 with a Supermicro MB using ATI SB700 chipset - which means it supports SATA 3.0.
Current storage is one o/s drive, and a 4-drive RAID10 array. The RAID10 is using the Linux softraid via mdadm. The drives are Seagates - ST3160811AS. So it's 320GB of storage, and SATA 1.5.
I'm seeing some warnings & errors in my logs & dmesg - and Google tells me this can the result of several factors, including the hard drives. I haven't seen any SMART warnings as yet - but I am getting a little nervous and thinking about upgrading the storage.
At this time, I'm just interested in recommendations for hard drives & partitioning strategy. At the time I created the RAID10 array - I was still in the early stages of learning these technologies (not that I know that much more now!) so I probably didn't take advantage of any of the advanced settings such as stripe size, cluster size, etc. I am using XFS.
As part of the potential upgrade, I'm considering changing to RAID6 - seems a bit more efficient use of space. I see no reason for SSD - I think a set of reasonable 7200rpm drives should be just fine. What I don't know is, compared to my current 4 drive RAID10 with SATA 1.5, would even a single SATA 3 drive be comparable in terms of seek performance? Should I stick with the RAID-10?
Daniel
SATA 1 vs SATA 3 won't change the seek performance of the drive,
unless your stay fully within the drives cache.
Only 2 things affect how many iops you can get from a drive. The
physical diameter and the speed. You can affect the physical diameter
of the drive by selecting how much of the drive you use, like only
using 200gigs out of a 500gig drive.
If you do opt for raid6, that would be a huge cpu overhead, and if you
stick with 4 drives, not worth it, every write would hit atleast 3
drives then, instead of just two they do currently, with no gain of
space.
Quoting "Daniel L. Miller" dmiller@amfes.com:
A little OT - but I've seen a few opinions voiced here by various
admins and I'd like to benefit.Currently running a single combined server for multiple operations -
fileserver, mailserver, webserver, virtual server, and whatever else
pops up. Current incarnation of the machine, after the last
rebuild, is an AMD Opteron 4180 with a Supermicro MB using ATI SB700
chipset - which means it supports SATA 3.0.Current storage is one o/s drive, and a 4-drive RAID10 array. The
RAID10 is using the Linux softraid via mdadm. The drives are
Seagates - ST3160811AS. So it's 320GB of storage, and SATA 1.5.I'm seeing some warnings & errors in my logs & dmesg - and Google
tells me this can the result of several factors, including the hard
drives. I haven't seen any SMART warnings as yet - but I am getting
a little nervous and thinking about upgrading the storage.At this time, I'm just interested in recommendations for hard drives
& partitioning strategy. At the time I created the RAID10 array - I
was still in the early stages of learning these technologies (not
that I know that much more now!) so I probably didn't take advantage
of any of the advanced settings such as stripe size, cluster size,
etc. I am using XFS.As part of the potential upgrade, I'm considering changing to RAID6
- seems a bit more efficient use of space. I see no reason for SSD
I think a set of reasonable 7200rpm drives should be just fine.
Daniel
What I don't know is, compared to my current 4 drive RAID10 with
SATA 1.5, would even a single SATA 3 drive be comparable in terms of
seek performance? Should I stick with the RAID-10?
On 8/30/2011 5:43 PM, Daniel L. Miller wrote:
A little OT - but I've seen a few opinions voiced here by various admins and I'd like to benefit.
RAID-10 is fine (note that the default mdadm RAID10 isn't actually RAID10, but it works well enough). RAID-6 won't be faster (and will probably be worse) although RAID-6 does do a bit better in a double-drive failure over RAID-10. The only way to get more performance out of (4) drives is to switch to 10k or 15k SAS (or SSDs).
For more information - see the Linux RAID mailing list: http://vger.kernel.org/majordomo-info.html
One problematic issue with consumer-grade SATA drives (which may or may not bite you) is that they will not time out on errors fast enough to keep mdadm happy. The "enterprise" grade drives are better about this (such as the ES.2 series), but for smaller arrays (6 drives or less) it's not as big of a deal. For bigger arrays, it's a definite issue, especially if you try and do RAID-6 over 8+ drives.
If you're getting SMART errors, then it's time to swap the drives out. If mdadm is reporting sync errors or dropping drives from the array, then get your backups squared away ASAP before fiddling.
My knee-jerk reaction when I hear 4-drive RAID-10 is that it has no hot-spare. Which means that as soon as 1 drive fails you're in dangerous territory (make sure it pages you automatically) since the array can't automatically repair. Make sure you can properly identify the drive that fails (via the serial numbers) and don't try a hot-swap.
(Take a look at /dev/disk/by-id, /dev/disk/by-uuid, etc. Export a copy of that information on a daily/weekly basis off of the machine. In a software RAID environment, it gives you better information about which drive serial # failed rather then relying on lights.)
Our mail server is 3-way RAID1 (triple mirror) for the OS and mail queue with a 5-disk RAID-10 (4+spare) for mail storage.
On 8/31/2011 12:41 PM, Thomas Harold wrote:
On 8/30/2011 5:43 PM, Daniel L. Miller wrote:
A little OT - but I've seen a few opinions voiced here by various admins and I'd like to benefit.
RAID-10 is fine (note that the default mdadm RAID10 isn't actually RAID10, but it works well enough). RAID-6 won't be faster (and will probably be worse) although RAID-6 does do a bit better in a double-drive failure over RAID-10. The only way to get more performance out of (4) drives is to switch to 10k or 15k SAS (or SSDs).
[...]
(Take a look at /dev/disk/by-id, /dev/disk/by-uuid, etc. Export a copy of that information on a daily/weekly basis off of the machine. In a software RAID environment, it gives you better information about which drive serial # failed rather then relying on lights.)
Our mail server is 3-way RAID1 (triple mirror) for the OS and mail queue with a 5-disk RAID-10 (4+spare) for mail storage.
Given my extensive requirements - I haven't yet filled my existing 320GB
- size isn't a big deal. Am I actually deriving much benefit from 4-disk RAID10 using 160GB discs - vs a 2-4 disc 1TB RAID1 array?
-- Daniel
On 9/1/2011 12:48 PM, Daniel L. Miller wrote:
Given my extensive requirements - I haven't yet filled my existing 320GB
- size isn't a big deal. Am I actually deriving much benefit from 4-disk RAID10 using 160GB discs - vs a 2-4 disc 1TB RAID1 array?
A pair of RAID-1 mirrors:
- easy to deal with
- you can attempt to manually balance load between the two arrays (storage on one pair, indexes and mail queue on other pair)
- disks can be pulled and taken to another machine and read one by one
- slightly harder to screw up (but both setups die if the wrong 2 disks fail)
RAID-10 over 4 disks:
- generally faster seeks
- generally faster read/write speeds due to striping
- generally the better choice for performance
- a bit harder to bury the disks vs a pair of mirrors
- lets you have a bigger partition
- all the eggs in a single array
If you're having performance problems on the existing RAID-10, your only real choices are to throw more spindles at it (move to a 6 or 8 disk RAID-10 w/ a hot-spare disk), throw faster spindles at it (10k/15k SAS), or move to SSD.
So, if you think you can manually balance the needs of the system, you could try a pair of independent mirrors. But if you want less hassle, stick with the RAID-10.
(And look into a tool like "atop" which can be run in the terminal and does a decent job of showing you whether the CPU/DISK is overly busy.)
Ditto,
Don't know anything on RAID 10 with four disks, but I agree with the two-disk scenario.
s.
"I merely function as a channel that filters music through the chaos of noise"
- Vangelis
From: Thomas Harold thomas-lists@nybeta.com To: dovecot@dovecot.org Sent: Friday, 2 September 2011, 3:23 Subject: Re: [Dovecot] OT - small hd recommendation
On 9/1/2011 12:48 PM, Daniel L. Miller wrote:
Given my extensive requirements - I haven't yet filled my existing 320GB
- size isn't a big deal. Am I actually deriving much benefit from 4-disk RAID10 using 160GB discs - vs a 2-4 disc 1TB RAID1 array?
A pair of RAID-1 mirrors:
- easy to deal with
- you can attempt to manually balance load between the two arrays (storage on one pair, indexes and mail queue on other pair)
- disks can be pulled and taken to another machine and read one by one
- slightly harder to screw up (but both setups die if the wrong 2 disks fail)
RAID-10 over 4 disks:
- generally faster seeks
- generally faster read/write speeds due to striping
- generally the better choice for performance
- a bit harder to bury the disks vs a pair of mirrors
- lets you have a bigger partition
- all the eggs in a single array
If you're having performance problems on the existing RAID-10, your only real choices are to throw more spindles at it (move to a 6 or 8 disk RAID-10 w/ a hot-spare disk), throw faster spindles at it (10k/15k SAS), or move to SSD.
So, if you think you can manually balance the needs of the system, you could try a pair of independent mirrors. But if you want less hassle, stick with the RAID-10.
(And look into a tool like "atop" which can be run in the terminal and does a decent job of showing you whether the CPU/DISK is overly busy.)
Hi,
At least, rather than Seagate AS series or some "green" series try using Seagate ST.SV series SATA HDDs, the 10$ difference is woth it. They are certified for 7/24 operation and will not fail on you as quickly. And they perform a little bit faster.
Regards, Kerem
On Fri, Sep 2, 2011 at 8:55 AM, Spyros Tsiolis stsiol@yahoo.co.uk wrote:
Ditto,
Don't know anything on RAID 10 with four disks, but I agree with the two-disk scenario.
s.
"I merely function as a channel that filters music through the chaos of noise"
- Vangelis
From: Thomas Harold thomas-lists@nybeta.com To: dovecot@dovecot.org Sent: Friday, 2 September 2011, 3:23 Subject: Re: [Dovecot] OT - small hd recommendation
On 9/1/2011 12:48 PM, Daniel L. Miller wrote:
Given my extensive requirements - I haven't yet filled my existing 320GB
- size isn't a big deal. Am I actually deriving much benefit from 4-disk RAID10 using 160GB discs - vs a 2-4 disc 1TB RAID1 array?
A pair of RAID-1 mirrors:
- easy to deal with
- you can attempt to manually balance load between the two arrays (storage on one pair, indexes and mail queue on other pair)
- disks can be pulled and taken to another machine and read one by one
- slightly harder to screw up (but both setups die if the wrong 2 disks fail)
RAID-10 over 4 disks:
- generally faster seeks
- generally faster read/write speeds due to striping
- generally the better choice for performance
- a bit harder to bury the disks vs a pair of mirrors
- lets you have a bigger partition
- all the eggs in a single array
If you're having performance problems on the existing RAID-10, your only real choices are to throw more spindles at it (move to a 6 or 8 disk RAID-10 w/ a hot-spare disk), throw faster spindles at it (10k/15k SAS), or move to SSD.
So, if you think you can manually balance the needs of the system, you could try a pair of independent mirrors. But if you want less hassle, stick with the RAID-10.
(And look into a tool like "atop" which can be run in the terminal and does a decent job of showing you whether the CPU/DISK is overly busy.)
-- Kerem Erciyes - Sistem Danismani http://keremerciyes.com
On 8/31/11, Daniel L. Miller dmiller@amfes.com wrote:
I'm seeing some warnings & errors in my logs & dmesg - and Google tells me this can the result of several factors, including the hard drives. I haven't seen any SMART warnings as yet - but I am getting a little nervous and thinking about upgrading the storage.
Possible to post some of those error messages?
As part of the potential upgrade, I'm considering changing to RAID6 - seems a bit more efficient use of space. I see no reason for SSD - I
It might be more efficient but you'll take a 3x IOPS penalty for writes vs RAID 1/10. Depending on your exact workload, it is most likely going to be a bad idea since you mention that you don't actually need more space.
think a set of reasonable 7200rpm drives should be just fine. What I don't know is, compared to my current 4 drive RAID10 with SATA 1.5, would even a single SATA 3 drive be comparable in terms of seek performance? Should I stick with the RAID-10?
Seek performance is largely a matter of I/O latency, SATA 3 only gives you more bandwidth to play with. A SATA 3 4200RPM HDD will seek slower than a SATA 1 7200RPM HDD.
For random I/O, more spindles/drives are usually the way to deal with it, like others suggested, splitting your drives into RAID 1 nodes and balancing different workloads on each node is probably your best bet if you don't mind doing it manually.
participants (6)
-
Daniel L. Miller
-
Emmanuel Noobadmin
-
Kerem Erciyes
-
Patrick Domack
-
Spyros Tsiolis
-
Thomas Harold