[Dovecot] IO rate quotas?

[Stan Hoeppner]

Eric Shubert put forth on 4/8/2011 12:03 PM:
> On 04/08/2011 08:58 AM, Stan Hoeppner wrote:
>> Eric Shubert put forth on 4/7/2011 9:54 PM:
>>
>>> Thanks for the great explanation, Stan.
>>>
>>> Just to clarify, did you mean to say that the former is cheaper in $$?
>>
>> Yes.  I just noticed that error and cringed.  :(  I did indeed mean to
>> say FORMER.  With a caveat however:  only if the necessary traffic
>> shaping can be accomplished with FOSS running on the Dovecot server and
>> the man hours to implement and test it are relatively low.
>>
>> If total man hours are more than a couple of days, or you end up needing
>> another box for shaping duty, or worse a commercial vendor solution,
>> then simply adding spindles to the current server may be cheaper
>> overall, and would offer additional benefits (speed/capacity increase)
>> for all users as well.
>>
> 
> I would hope that traffic shaping could be done in an affordable manner,
> like you say with FOSS on the Dovecot server.

>From Javier's post it would seem this solution will indeed be man hour
intensive. :(

> As Timo pointed out, there might be some other tuning that needs to be
> addressed as well. This could be as simple as changing the elevator, or
> doing some of the other 4 things you mentioned previously.

Choice of elevator is as much dependent upon the block path hardware as
the workload, maybe more so.  Deadline works well for single disks and
mdraid, but with many caching hardware RAID controllers and with FC or
iSCSI SAN controllers, noop will often outperform deadline.  I think
everyone "in the know" would agree that CFQ sucks pretty much across the
board.  Thus, always test each elevator in a production setting and
gather real metrics.  Then select the overall best performer.  In some
cases all of them may perform equally well.  The greater the load, the
more dramatic the differences.

> BL, one client should not be able to bring a server to its knees. ;)

I don't believe this is what is happening.  It appears to me that the
OP's Dovecot server is likely "close to the edge", and when the
described scenario occurs, the rouge IMAP client is simply pushing the
host off the cliff, causing _everything_ to come to a crawl: the server,
the clients, and the rogue client.  Given that he's seeing load of 50
there's probably more going on than what we've been told.  The host is
likely going heavily into swap, and/or the disk subsystem is simply not
sized to carry the extra IOPS demanded.  If both are true you have a
"perfect storm" hitting the block IO hardware.

To playfully (respectfully) contradict the notion that one client should
not be able to bring a server to its knees, take one client and perform
a full text search for "sualcatnas" of a 5GB shared IMAP folder and sub
folders residing on maildir storage.  Do this on a dual core production
server w/4GB RAM and 4 disks in hardware RAID10, i.e. a low end system
that normally serves 50-150 users, during peak normal usage--first thing
in the morning or right after lunch.

I think you'll find that this one client operation can definitely bring
the server's IO subsystem to its knees for a short time, or at minimum
increase response times of other clients to a very conspicuous degree.
FTS causes massive head seeking with maildir storage, and if the disk
subsystem isn't sized accordingly, knees hit the pavement.

Anyway, I think we really need more data from the OP to enable us to
give definitive advice.

-- 
Stan