[Dovecot] Better to use a single large storage server or multiple smaller for mdbox?

[Ed W]

On 14/04/2012 04:31, Stan Hoeppner wrote:
> On 4/13/2012 10:31 AM, Ed W wrote:
>> On 13/04/2012 13:33, Stan Hoeppner wrote:
>>> In closing, I'll simply say this:  If hardware, whether a mobo-down SATA
>>> chip, or a $100K SGI SAN RAID controller, allowed silent data corruption
>>> or transmission to occur, there would be no storage industry, and we'll
>>> all still be using pen and paper.  The questions you're asking were
>>> solved by hardware and software engineers decades ago.  You're fretting
>>> and asking about things that were solved decades ago.
>> So why are so many people getting excited about it now?
> "So many"?  I know of one person "getting excited" about it.

You love being vague don't you?  Go on, I'll bite again, do you mean 
yourself?

:-)

> Data densities and overall storage sizes and complexity at the top end
> of the spectrum are increasing at a faster rate than the
> consistency/validation mechanisms.  That's the entire point of the
> various academic studies on the issue.

Again, you love being vague.  By your dismissive "academic studies" 
phrase, do you mean studies done on a major industrial player, ie NetApp 
in this case?  Or do you mean that it's rubbish because they asked 
someone with some background in statistics to do the work, rather than 
asking someone sitting nearby in the office to do it?

I don't think the researcher broke into NetApp to do this research, so 
we have to conclude that the industrial partner was onboard.  NetApp 
seem to do a bunch of engineering of their own (got enough patents..) 
that I think we can safely assume they very much do their own research 
on this and it's not just "academic"...  I doubt they publish all their 
own internal research, be thankful you got to see some of the results 
this way...

>    Note that the one study required
> a sample set of 1.5 million disk drives.  If the phenomenon were a
> regular occurrence as you would have everyone here believe, they could
> have used a much smaller sample set.

Sigh... You could criticise the study if it had a small number of drives 
as being under-representive and now you criticise a large study for 
having too many observations...

You cannot have "too many" observations when measuring a small and 
unpredictable phenomena...

Where does it say that they could NOT have reproduced this study with 
just 10 drives?  If you have 1.5 million available, why not use all the 
results??


> Ed, this is an academic exercise.  Academia leads industry.  Almost
> always has.  Academia blows the whistle and waves hands, prompting
> industry to take action.

Sigh... We are back to the start of the email thread again... Gosh you 
seem to love arguing and muddying the water for zero reason but to have 
the last word?

It's *trivial* to do a google search and hit *lots* of reports of 
corruptions in various parts of the system, from corrupting drivers, to 
hardware which writes incorrectly, to operating system flaws.  I just 
found a bunch more in the Redhat database today while looking for 
something else.  You yourself are very vocal on avoiding certain brands 
of HD controller which have been rumoured to cause corrupted data... 
(and thankyou for revealing that kind of thing - it's very helpful)

Don't veer off at a tangent now: The *original* email this has spawned 
is about a VERY specific point.  RAID1 appears to offer less protection 
against a class of error conditions than does RAID6.  Nothing more, 
nothing less.  Don't veer off and talk about the minutiae of testing 
studies at universities, this is a straightforward claim that you have 
been jumping around and avoiding answering with claims of needing to 
educate me on SCSI protocols and other fatuous responses. Nor deviate 
and discuss that RAID6 is inappropriate for many situations - we all get 
that...



> There is nothing normal users need to do to address this problem.

...except sit tight and hope they don't loose anything important!

:-)


> Having the prestigious degree that you do, you should already understand
> the relationship between academic research and industry, and the
> considerable lead times involved.

I'm guessing you haven't attended higher education then?  You are 
confusing graduate and post-graduate systems...

Byee

Ed W