[Dovecot] IO rate quotas?
Hi
Okay so we've been having this issue since forever and I figured why the heck not ask it here since it's mostly related to dovecot. The thing is, we have a huge amount of public folders (at the moment, around 1100). Now, with dovecot indexing and caching we're mostly okay, also being careful with things like Thunderbird 3's default 'Download everything' option ("Keep messages for this account on this computer") and such. However once in a while, someone goes rogue, we install a new version of thunderbird, someone accidentally sets up an email account in thunderbird mistakingly leaving the download everything option on. This causes high IO on our server, and a single user can quickly more or less kill the whole server in mere minutes, load average quickly spiking to 30-40-50 and everything becomes sloooooow (obviously).
Is there any way I can limit the amount of IO a certain user can use?
Thanks and best regards, KT
P.S.
my dovecot -n is:
# 2.0.9: /usr/local/etc/dovecot/dovecot.conf # OS: Linux 2.6.26-2-xen-amd64 x86_64 Debian 5.0.8 auth_mechanisms = plain login disable_plaintext_auth = no listen = * mail_location = maildir:~/Maildir mail_plugins = acl maildir_very_dirty_syncs = yes namespace { inbox = yes location = prefix = separator = . type = private } namespace { location = maildir:/home/_shared/projects:INDEX=~/Maildir/_shared prefix = shared. separator = . subscriptions = no type = public } namespace { location = maildir:/home/_shared/sys:INDEX=~/Maildir/sys prefix = sys. separator = . subscriptions = no type = public } passdb { driver = pam } plugin { acl = vfile mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename flag_change append mail_log_fields = uid box msgid from subject size vsize flags mail_log_group_events = no } protocols = imap pop3 service imap { executable = imap } ssl_cert = </etc/ssl/certs/dovecot.pem ssl_key = </etc/ssl/private/dovecot.pem userdb { driver = passwd } verbose_proctitle = yes protocol imap { mail_max_userip_connections = 40 mail_plugins = acl imap_acl }
Kádár Tamás (KTamas) put forth on 4/7/2011 9:33 AM:
Hi
Okay so we've been having this issue since forever and I figured why the heck not ask it here since it's mostly related to dovecot. The thing is, we have a huge amount of public folders (at the moment, around 1100). Now, with dovecot indexing and caching we're mostly okay, also being careful with things like Thunderbird 3's default 'Download everything' option ("Keep messages for this account on this computer") and such. However once in a while, someone goes rogue, we install a new version of thunderbird, someone accidentally sets up an email account in thunderbird mistakingly leaving the download everything option on. This causes high IO on our server, and a single user can quickly more or less kill the whole server in mere minutes, load average quickly spiking to 30-40-50 and everything becomes sloooooow (obviously).
Is there any way I can limit the amount of IO a certain user can use?
TTBOMK Dovecot has no data rate limiting controls, neither globally nor per user. As I see it you have a few of options:
- Switch to mdbox or mbox storage format to decrease IOs per email
- Beef up the server with many more RAID spindles
- Eliminate all unnecessary filesystem metadata and logging activity for instance, atime, if not done already
- Switch to a filesystem matched to your workload EXT3/4/Reiser are not optimal for high concurrency multi-user server workloads switch to XFS or JFS if currently using EXT3/4
- Install a traffic shaper in front of, or on, the Dovecot server Configure it to clamp any TCP sessions that exceed, say, 10 Mbit/s for more than 10 seconds down to 1 Mbit/s for a 1 minute duration. Tune to taste until you get the desired results.
Option 5 is the most sure fire way to solve the problem you describe. It is the most direct solution, and likely free (not including your time to set it up), assuming netfilter etc scripts are available to accomplish this.
-- Stan
On 04/07/2011 12:45 PM, Stan Hoeppner wrote:
Kádár Tamás (KTamas) put forth on 4/7/2011 9:33 AM:
Hi
Okay so we've been having this issue since forever and I figured why the heck not ask it here since it's mostly related to dovecot. The thing is, we have a huge amount of public folders (at the moment, around 1100). Now, with dovecot indexing and caching we're mostly okay, also being careful with things like Thunderbird 3's default 'Download everything' option ("Keep messages for this account on this computer") and such. However once in a while, someone goes rogue, we install a new version of thunderbird, someone accidentally sets up an email account in thunderbird mistakingly leaving the download everything option on. This causes high IO on our server, and a single user can quickly more or less kill the whole server in mere minutes, load average quickly spiking to 30-40-50 and everything becomes sloooooow (obviously).
Is there any way I can limit the amount of IO a certain user can use?
TTBOMK Dovecot has no data rate limiting controls, neither globally nor per user. As I see it you have a few of options:
- Switch to mdbox or mbox storage format to decrease IOs per email
- Beef up the server with many more RAID spindles
- Eliminate all unnecessary filesystem metadata and logging activity for instance, atime, if not done already
- Switch to a filesystem matched to your workload EXT3/4/Reiser are not optimal for high concurrency multi-user server workloads switch to XFS or JFS if currently using EXT3/4
- Install a traffic shaper in front of, or on, the Dovecot server Configure it to clamp any TCP sessions that exceed, say, 10 Mbit/s for more than 10 seconds down to 1 Mbit/s for a 1 minute duration. Tune to taste until you get the desired results.
Option 5 is the most sure fire way to solve the problem you describe. It is the most direct solution, and likely free (not including your time to set it up), assuming netfilter etc scripts are available to accomplish this.
I liked option 5 as being the best choice for this, at first.
Then I began to wonder, when a client does this, is there a process created for each mail directory? I'm not familiar enough with the IMAP protocol or dovecot to know off hand. If so though, could that be a problem?
Expecting that Timo has probably addressed this already (he's very thorough to say the least), I looked through the (v2.0.11) configuration parameters, and found: #mail_max_userip_connections = 10 in the 20-imap.conf file. This leads me to believe that yes, there can reasonably be multiple processes per user. With 1100 folders to download, I can see where the downloading of all that might overload a system. I would guess that reducing this value somewhat would tend to limit the impact any one user could have on the system as a whole. At the same time however, overall performance may suffer the lower this value is.
BL, your bottleneck (high load) might be more related to the number of processes the user has at once, more so than the i/o or bandwidth demands they're causing. Reduce the # of processes, and the others go down too.
As a quick and easy remedy, I think I'd try adjusting this value some.
-- -Eric 'shubes'
Eric Shubert put forth on 4/7/2011 4:04 PM:
On 04/07/2011 12:45 PM, Stan Hoeppner wrote:
Kádár Tamás (KTamas) put forth on 4/7/2011 9:33 AM:
Hi
Okay so we've been having this issue since forever and I figured why the heck not ask it here since it's mostly related to dovecot. The thing is, we have a huge amount of public folders (at the moment, around 1100). Now, with dovecot indexing and caching we're mostly okay, also being careful with things like Thunderbird 3's default 'Download everything' option ("Keep messages for this account on this computer") and such. However once in a while, someone goes rogue, we install a new version of thunderbird, someone accidentally sets up an email account in thunderbird mistakingly leaving the download everything option on. This causes high IO on our server, and a single user can quickly more or less kill the whole server in mere minutes, load average quickly spiking to 30-40-50 and everything becomes sloooooow (obviously).
Is there any way I can limit the amount of IO a certain user can use?
TTBOMK Dovecot has no data rate limiting controls, neither globally nor per user. As I see it you have a few of options:
- Switch to mdbox or mbox storage format to decrease IOs per email
- Beef up the server with many more RAID spindles
- Eliminate all unnecessary filesystem metadata and logging activity for instance, atime, if not done already
- Switch to a filesystem matched to your workload EXT3/4/Reiser are not optimal for high concurrency multi-user server workloads switch to XFS or JFS if currently using EXT3/4
- Install a traffic shaper in front of, or on, the Dovecot server Configure it to clamp any TCP sessions that exceed, say, 10 Mbit/s for more than 10 seconds down to 1 Mbit/s for a 1 minute duration. Tune to taste until you get the desired results.
Option 5 is the most sure fire way to solve the problem you describe. It is the most direct solution, and likely free (not including your time to set it up), assuming netfilter etc scripts are available to accomplish this.
I liked option 5 as being the best choice for this, at first.
Then I began to wonder, when a client does this, is there a process created for each mail directory? I'm not familiar enough with the IMAP protocol or dovecot to know off hand. If so though, could that be a problem?
This is irrelevant WRT traffic shaping. Traffic shaping simply counts IP packets per unit time for a packet matching source+destination address combo X. When the defined threshold is exceeded, future X packets are delayed Y seconds to achieve max data rate Z. This occurs many layers below IMAP in the OSI model, thus the action is transparent to Dovecot. In layman's terms, this is akin to sticking the offending client PC on the international space station for a minute at a time. The dramatically increased packet latency causes a dramatic drop in throughput, thus a dramatically decreased IO load on the server.
Expecting that Timo has probably addressed this already (he's very thorough to say the least), I looked through the (v2.0.11) configuration parameters, and found: #mail_max_userip_connections = 10
This exists in 1.x as well, but it doesn't apply to this situation.
in the 20-imap.conf file. This leads me to believe that yes, there can reasonably be multiple processes per user. With 1100 folders to download, I can see where the downloading of all that might overload a system. I would guess that reducing this value somewhat would tend to limit the impact any one user could have on the system as a whole. At the same time however, overall performance may suffer the lower this value is.
Last I checked, by default, TBird opens 5 IMAP connections per session. In my experience, four of the five go unused 99.999% of the time. Only one is used for over 99.999% of IMAP traffic. AIUI, the extra 4 are used primarily for things like IMAP IDLE notifications on non-selected folders. A quick glance at the TIME+ column in top will demonstrate this. Thus I've been configuring TBird clients to use only 1 cached connection for quite some time.
This doesn't limit client bandwidth or minimize server load, but it cuts down on sleeping processes by a factor of 5, and saves a small amount of memory. I say small because Linux only stores binary code once in memory no matter how many processes you have running. Also, all the IMAP processes will be sharing buffer cache. Thus, you don't really save significant memory by cutting the IMAP processes by 5x, but I find managing one IMAP process per user much more elegant than 5 processes per user.
BL, your bottleneck (high load) might be more related to the number of processes the user has at once, more so than the i/o or bandwidth demands they're causing. Reduce the # of processes, and the others go down too.
This is not correct. Number of same user IMAP processes has no bearing on the OP's issue. See above. Email, whether SMTP or IMAP, is disk seek bound, rarely, if ever, CPU or memory bound, unless a server machine is simply horribly architected or running insanely complex sieve rules etc.
The OP's problem is that a single client PC attempts to download the entire 1100 public folder set and a single user mailbox, apparently many hundreds of MBs or GBs in size. Over a GbE link this could be as much as 50 to 90 MB/s. As the data being read from disk is not sequential, this generates a substantial random IO load on the server, exacerbated by the Maildir storage format, requiring one file read per message.
The OP is simply running out of disk IOPS in this scenario. The two sledgehammer methods for fixing this are rate limiting the client at the packet level, which in turn reduces the disk seek load, or substantially beefing up the disk subsystem. The latter is cheaper (in $$), more direct, and more predictable. As my address RHS would suggest, I'm all for beefing up hardware. But in this case, it's not the optimal solution.
-- Stan
On 04/07/2011 06:49 PM, Stan Hoeppner wrote:
Eric Shubert put forth on 4/7/2011 4:04 PM:
On 04/07/2011 12:45 PM, Stan Hoeppner wrote:
Kádár Tamás (KTamas) put forth on 4/7/2011 9:33 AM:
Hi
Okay so we've been having this issue since forever and I figured why the heck not ask it here since it's mostly related to dovecot. The thing is, we have a huge amount of public folders (at the moment, around 1100). Now, with dovecot indexing and caching we're mostly okay, also being careful with things like Thunderbird 3's default 'Download everything' option ("Keep messages for this account on this computer") and such. However once in a while, someone goes rogue, we install a new version of thunderbird, someone accidentally sets up an email account in thunderbird mistakingly leaving the download everything option on. This causes high IO on our server, and a single user can quickly more or less kill the whole server in mere minutes, load average quickly spiking to 30-40-50 and everything becomes sloooooow (obviously).
Is there any way I can limit the amount of IO a certain user can use?
TTBOMK Dovecot has no data rate limiting controls, neither globally nor per user. As I see it you have a few of options:
- Switch to mdbox or mbox storage format to decrease IOs per email
- Beef up the server with many more RAID spindles
- Eliminate all unnecessary filesystem metadata and logging activity for instance, atime, if not done already
- Switch to a filesystem matched to your workload EXT3/4/Reiser are not optimal for high concurrency multi-user server workloads switch to XFS or JFS if currently using EXT3/4
- Install a traffic shaper in front of, or on, the Dovecot server Configure it to clamp any TCP sessions that exceed, say, 10 Mbit/s for more than 10 seconds down to 1 Mbit/s for a 1 minute duration. Tune to taste until you get the desired results.
Option 5 is the most sure fire way to solve the problem you describe. It is the most direct solution, and likely free (not including your time to set it up), assuming netfilter etc scripts are available to accomplish this.
I liked option 5 as being the best choice for this, at first.
Then I began to wonder, when a client does this, is there a process created for each mail directory? I'm not familiar enough with the IMAP protocol or dovecot to know off hand. If so though, could that be a problem?
This is irrelevant WRT traffic shaping. Traffic shaping simply counts IP packets per unit time for a packet matching source+destination address combo X. When the defined threshold is exceeded, future X packets are delayed Y seconds to achieve max data rate Z. This occurs many layers below IMAP in the OSI model, thus the action is transparent to Dovecot. In layman's terms, this is akin to sticking the offending client PC on the international space station for a minute at a time. The dramatically increased packet latency causes a dramatic drop in throughput, thus a dramatically decreased IO load on the server.
Expecting that Timo has probably addressed this already (he's very thorough to say the least), I looked through the (v2.0.11) configuration parameters, and found: #mail_max_userip_connections = 10
This exists in 1.x as well, but it doesn't apply to this situation.
in the 20-imap.conf file. This leads me to believe that yes, there can reasonably be multiple processes per user. With 1100 folders to download, I can see where the downloading of all that might overload a system. I would guess that reducing this value somewhat would tend to limit the impact any one user could have on the system as a whole. At the same time however, overall performance may suffer the lower this value is.
Last I checked, by default, TBird opens 5 IMAP connections per session. In my experience, four of the five go unused 99.999% of the time. Only one is used for over 99.999% of IMAP traffic. AIUI, the extra 4 are used primarily for things like IMAP IDLE notifications on non-selected folders. A quick glance at the TIME+ column in top will demonstrate this. Thus I've been configuring TBird clients to use only 1 cached connection for quite some time.
This doesn't limit client bandwidth or minimize server load, but it cuts down on sleeping processes by a factor of 5, and saves a small amount of memory. I say small because Linux only stores binary code once in memory no matter how many processes you have running. Also, all the IMAP processes will be sharing buffer cache. Thus, you don't really save significant memory by cutting the IMAP processes by 5x, but I find managing one IMAP process per user much more elegant than 5 processes per user.
BL, your bottleneck (high load) might be more related to the number of processes the user has at once, more so than the i/o or bandwidth demands they're causing. Reduce the # of processes, and the others go down too.
This is not correct. Number of same user IMAP processes has no bearing on the OP's issue. See above. Email, whether SMTP or IMAP, is disk seek bound, rarely, if ever, CPU or memory bound, unless a server machine is simply horribly architected or running insanely complex sieve rules etc.
The OP's problem is that a single client PC attempts to download the entire 1100 public folder set and a single user mailbox, apparently many hundreds of MBs or GBs in size. Over a GbE link this could be as much as 50 to 90 MB/s. As the data being read from disk is not sequential, this generates a substantial random IO load on the server, exacerbated by the Maildir storage format, requiring one file read per message.
The OP is simply running out of disk IOPS in this scenario. The two sledgehammer methods for fixing this are rate limiting the client at the packet level, which in turn reduces the disk seek load, or substantially beefing up the disk subsystem. The latter is cheaper (in $$), more direct, and more predictable. As my address RHS would suggest, I'm all for beefing up hardware. But in this case, it's not the optimal solution.
Thanks for the great explanation, Stan.
Just to clarify, did you mean to say that the former is cheaper in $$?
-- -Eric 'shubes'
Eric Shubert put forth on 4/7/2011 9:54 PM:
Thanks for the great explanation, Stan.
Just to clarify, did you mean to say that the former is cheaper in $$?
Yes. I just noticed that error and cringed. :( I did indeed mean to say FORMER. With a caveat however: only if the necessary traffic shaping can be accomplished with FOSS running on the Dovecot server and the man hours to implement and test it are relatively low.
If total man hours are more than a couple of days, or you end up needing another box for shaping duty, or worse a commercial vendor solution, then simply adding spindles to the current server may be cheaper overall, and would offer additional benefits (speed/capacity increase) for all users as well.
-- Stan
On 04/08/2011 08:58 AM, Stan Hoeppner wrote:
Eric Shubert put forth on 4/7/2011 9:54 PM:
Thanks for the great explanation, Stan.
Just to clarify, did you mean to say that the former is cheaper in $$?
Yes. I just noticed that error and cringed. :( I did indeed mean to say FORMER. With a caveat however: only if the necessary traffic shaping can be accomplished with FOSS running on the Dovecot server and the man hours to implement and test it are relatively low.
If total man hours are more than a couple of days, or you end up needing another box for shaping duty, or worse a commercial vendor solution, then simply adding spindles to the current server may be cheaper overall, and would offer additional benefits (speed/capacity increase) for all users as well.
I would hope that traffic shaping could be done in an affordable manner, like you say with FOSS on the Dovecot server.
As Timo pointed out, there might be some other tuning that needs to be addressed as well. This could be as simple as changing the elevator, or doing some of the other 4 things you mentioned previously.
BL, one client should not be able to bring a server to its knees. ;)
-- -Eric 'shubes'
I would hope that traffic shaping could be done in an affordable manner, like you say with FOSS on the Dovecot server.
Go to www.lartc.org (linux advanced routing & traffic control). They have a lot of doc and a great mailing list
Traffic shaping is a bit tricky until you understand all the mess, but when you "get it" is a very powerfull tool.
Regards
Javier
Eric Shubert put forth on 4/8/2011 12:03 PM:
On 04/08/2011 08:58 AM, Stan Hoeppner wrote:
Eric Shubert put forth on 4/7/2011 9:54 PM:
Thanks for the great explanation, Stan.
Just to clarify, did you mean to say that the former is cheaper in $$?
Yes. I just noticed that error and cringed. :( I did indeed mean to say FORMER. With a caveat however: only if the necessary traffic shaping can be accomplished with FOSS running on the Dovecot server and the man hours to implement and test it are relatively low.
If total man hours are more than a couple of days, or you end up needing another box for shaping duty, or worse a commercial vendor solution, then simply adding spindles to the current server may be cheaper overall, and would offer additional benefits (speed/capacity increase) for all users as well.
I would hope that traffic shaping could be done in an affordable manner, like you say with FOSS on the Dovecot server.
From Javier's post it would seem this solution will indeed be man hour intensive. :(
As Timo pointed out, there might be some other tuning that needs to be addressed as well. This could be as simple as changing the elevator, or doing some of the other 4 things you mentioned previously.
Choice of elevator is as much dependent upon the block path hardware as the workload, maybe more so. Deadline works well for single disks and mdraid, but with many caching hardware RAID controllers and with FC or iSCSI SAN controllers, noop will often outperform deadline. I think everyone "in the know" would agree that CFQ sucks pretty much across the board. Thus, always test each elevator in a production setting and gather real metrics. Then select the overall best performer. In some cases all of them may perform equally well. The greater the load, the more dramatic the differences.
BL, one client should not be able to bring a server to its knees. ;)
I don't believe this is what is happening. It appears to me that the OP's Dovecot server is likely "close to the edge", and when the described scenario occurs, the rouge IMAP client is simply pushing the host off the cliff, causing _everything_ to come to a crawl: the server, the clients, and the rogue client. Given that he's seeing load of 50 there's probably more going on than what we've been told. The host is likely going heavily into swap, and/or the disk subsystem is simply not sized to carry the extra IOPS demanded. If both are true you have a "perfect storm" hitting the block IO hardware.
To playfully (respectfully) contradict the notion that one client should not be able to bring a server to its knees, take one client and perform a full text search for "sualcatnas" of a 5GB shared IMAP folder and sub folders residing on maildir storage. Do this on a dual core production server w/4GB RAM and 4 disks in hardware RAID10, i.e. a low end system that normally serves 50-150 users, during peak normal usage--first thing in the morning or right after lunch.
I think you'll find that this one client operation can definitely bring the server's IO subsystem to its knees for a short time, or at minimum increase response times of other clients to a very conspicuous degree. FTS causes massive head seeking with maildir storage, and if the disk subsystem isn't sized accordingly, knees hit the pavement.
Anyway, I think we really need more data from the OP to enable us to give definitive advice.
-- Stan
Am 09.04.2011 00:25, schrieb Stan Hoeppner:
Eric Shubert put forth on 4/8/2011 12:03 PM:
On 04/08/2011 08:58 AM, Stan Hoeppner wrote:
Eric Shubert put forth on 4/7/2011 9:54 PM:
Thanks for the great explanation, Stan.
Just to clarify, did you mean to say that the former is cheaper in $$?
Yes. I just noticed that error and cringed. :( I did indeed mean to say FORMER. With a caveat however: only if the necessary traffic shaping can be accomplished with FOSS running on the Dovecot server and the man hours to implement and test it are relatively low.
If total man hours are more than a couple of days, or you end up needing another box for shaping duty, or worse a commercial vendor solution, then simply adding spindles to the current server may be cheaper overall, and would offer additional benefits (speed/capacity increase) for all users as well.
I would hope that traffic shaping could be done in an affordable manner, like you say with FOSS on the Dovecot server.
From Javier's post it would seem this solution will indeed be man hour intensive. :(
As Timo pointed out, there might be some other tuning that needs to be addressed as well. This could be as simple as changing the elevator, or doing some of the other 4 things you mentioned previously.
Choice of elevator is as much dependent upon the block path hardware as the workload, maybe more so. Deadline works well for single disks and mdraid, but with many caching hardware RAID controllers and with FC or iSCSI SAN controllers, noop will often outperform deadline.
just for info not to flame
however deadline sometimes is recommended for performance on some 3ware raid controllers, be aware you can run into troubles
https://www.3ware.com/3warekb/article.aspx?id=10889
try carefully !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
i recent found that with drbd ocfs2 and 3ware quiet nohz=off cpufreq=no elevator=deadline in grub is needed, by buggy 3ware ( elevator is optional but raises performance )
info
http://kerneltrap.org/mailarchive/linux-kernel/2010/9/21/4622280/thread
it took me weeks of debugging. so definite dont recommend some 3ware controllers at present
I think
everyone "in the know" would agree that CFQ sucks pretty much across the board. Thus, always test each elevator in a production setting and gather real metrics. Then select the overall best performer. In some cases all of them may perform equally well. The greater the load, the more dramatic the differences.
BL, one client should not be able to bring a server to its knees. ;)
I don't believe this is what is happening. It appears to me that the OP's Dovecot server is likely "close to the edge", and when the described scenario occurs, the rouge IMAP client is simply pushing the host off the cliff, causing _everything_ to come to a crawl: the server, the clients, and the rogue client. Given that he's seeing load of 50 there's probably more going on than what we've been told. The host is likely going heavily into swap, and/or the disk subsystem is simply not sized to carry the extra IOPS demanded. If both are true you have a "perfect storm" hitting the block IO hardware.
i found dovecot isnt so much in trouble with io as one might think with 3000 mailboxes on drbd ocfs2 3ware raid without deadline and heavy traffic but apache squirrelmail was nearly unusable without deadline having vhost and logging on drbd
To playfully (respectfully) contradict the notion that one client should not be able to bring a server to its knees, take one client and perform a full text search for "sualcatnas" of a 5GB shared IMAP folder and sub folders residing on maildir storage. Do this on a dual core production server w/4GB RAM and 4 disks in hardware RAID10, i.e. a low end system that normally serves 50-150 users, during peak normal usage--first thing in the morning or right after lunch.
agree never noticed one client getting the server into troubles
I think you'll find that this one client operation can definitely bring the server's IO subsystem to its knees for a short time, or at minimum increase response times of other clients to a very conspicuous degree. FTS causes massive head seeking with maildir storage, and if the disk subsystem isn't sized accordingly, knees hit the pavement.
Anyway, I think we really need more data from the OP to enable us to give definitive advice.
-- Best Regards
MfG Robert Schetterer
Germany/Munich/Bavaria
On Thu, 2011-04-07 at 16:33 +0200, Kádár Tamás (KTamas) wrote:
However once in a while, someone goes rogue, we install a new version of thunderbird, someone accidentally sets up an email account in thunderbird mistakingly leaving the download everything option on. This causes high IO on our server, and a single user can quickly more or less kill the whole server in mere minutes, load average quickly spiking to 30-40-50 and everything becomes sloooooow (obviously).
So a single process that is reading files fast enough from disk can cause disk IO to spike in a way that makes all other processes wait for available disk IO? I don't think that's a common problem. Typically only that one process would be waiting for disk IO and the load would increase by 1 (or maybe a little bit more, but definitely not by 50).. Maybe you should look into Linux I/O schedulers and see if you're using the wrong one.
Is there any way I can limit the amount of IO a certain user can use?
Not in Dovecot.
- Timo Sirainen <tss@iki.fi>:
On Thu, 2011-04-07 at 16:33 +0200, Kádár Tamás (KTamas) wrote:
However once in a while, someone goes rogue, we install a new version of thunderbird, someone accidentally sets up an email account in thunderbird mistakingly leaving the download everything option on. This causes high IO on our server, and a single user can quickly more or less kill the whole server in mere minutes, load average quickly spiking to 30-40-50 and everything becomes sloooooow (obviously).
So a single process that is reading files fast enough from disk can cause disk IO to spike in a way that makes all other processes wait for available disk IO? I don't think that's a common problem. Typically only that one process would be waiting for disk IO and the load would increase by 1 (or maybe a little bit more, but definitely not by 50).. Maybe you should look into Linux I/O schedulers and see if you're using the wrong one.
Sounds like a filesystem problem or a scheduler issue, yes.
Am 08.04.2011 15:53, schrieb Ralf Hildebrandt:
- Timo Sirainen <tss@iki.fi>:
On Thu, 2011-04-07 at 16:33 +0200, Kádár Tamás (KTamas) wrote:
However once in a while, someone goes rogue, we install a new version of thunderbird, someone accidentally sets up an email account in thunderbird mistakingly leaving the download everything option on. This causes high IO on our server, and a single user can quickly more or less kill the whole server in mere minutes, load average quickly spiking to 30-40-50 and everything becomes sloooooow (obviously).
So a single process that is reading files fast enough from disk can cause disk IO to spike in a way that makes all other processes wait for available disk IO? I don't think that's a common problem. Typically only that one process would be waiting for disk IO and the load would increase by 1 (or maybe a little bit more, but definitely not by 50).. Maybe you should look into Linux I/O schedulers and see if you're using the wrong one.
Sounds like a filesystem problem or a scheduler issue, yes.
perhaps try
echo "deadline" > /sys/block/sda/queue/scheduler
-- Best Regards
MfG Robert Schetterer
Germany/Munich/Bavaria
participants (7)
-
Eric Shubert
-
Javier de Miguel Rodriguez
-
Kádár Tamás (KTamas)
-
Ralf Hildebrandt
-
Robert Schetterer
-
Stan Hoeppner
-
Timo Sirainen