Re: [Dovecot] IO rate quotas?
From: Timo Sirainen <tss@iki.fi> Subject: Re: [Dovecot] IO rate quotas?
So a single process that is reading files fast enough from disk can cause disk IO to spike in a way that makes all other processes wait for available disk IO?
It's not a single process.
Thunderbird's newest defaults are to syncronize every folder and it will open one connectrion per folder in order to do that.
Appropriate disk/swappiness/readahead/cache tuning is required - along with enough ram to cope with several users trying this at once.
The underlaying filesystem is also important. as a for-instance, this kind of load with maildir format will effectively _kill_ a GFS box.
Our fix was to install systemwide defaults on managed systems to not synchronize (not needed, the home directories are on network storage anyway) and to ask people to check/unset this on laptops/etc
Wow, thanks everyone for the replies.
Already using XFS as a filesystem with noatime. Switching to faster disks/raids/etc is not easy, we rent hosted server in Germany, to make real changes i'd have to get my own server etc.
I'll play around with disk schedulers and report back.
Thanks again for all the help.
On Mon, Apr 11, 2011 at 12:58 PM, Alan Brown <ajb2@mssl.ucl.ac.uk> wrote:
From: Timo Sirainen <tss@iki.fi> Subject: Re: [Dovecot] IO rate quotas?
So a single process that is reading files fast enough from disk can cause disk IO to spike in a way that makes all other processes wait for available disk IO?
It's not a single process.
Thunderbird's newest defaults are to syncronize every folder and it will open one connectrion per folder in order to do that.
Appropriate disk/swappiness/readahead/cache tuning is required - along with enough ram to cope with several users trying this at once.
The underlaying filesystem is also important. as a for-instance, this kind of load with maildir format will effectively _kill_ a GFS box.
Our fix was to install systemwide defaults on managed systems to not synchronize (not needed, the home directories are on network storage anyway) and to ask people to check/unset this on laptops/etc
Kádár Tamás (KTamas) put forth on 4/11/2011 12:20 PM:
Wow, thanks everyone for the replies.
Already using XFS as a filesystem with noatime. Switching to faster disks/raids/etc is not easy, we rent hosted server in Germany, to make real changes i'd have to get my own server etc.
Use relatime (the default) instead of noatime: http://xfs.org/index.php/XFS_FAQ#Q:_Is_using_noatime_or.2Fand_nodiratime_at_...
Far more importantly, using kernel 2.6.36+ with the 'delaylog' mount option can increase XFS metadata write performance by a factor of 10 to 100+ on metadata write heavy systems. If you're not already using this, update your kernel to 2.6.37.6 and mount your XFS filesystems with delaylog. Doing this is free but for your time and reboot downtime, and could land you a huge decrease in metadata IOPS hitting the physical storage, in essence giving you a free hardware upgrade.
I'll play around with disk schedulers and report back.
As I mentioned previously, changing the elevator won't address the problem if you have real hardware RAID with decent sized write cache (512MB or more) either PCIe HBA or FC/iSCSI SAN array. In the case of hardware RAID, use the 'noop' elevator.
-- Stan
On 04/11/2011 03:58 AM, Alan Brown wrote:
From: Timo Sirainen <tss@iki.fi> Subject: Re: [Dovecot] IO rate quotas?
So a single process that is reading files fast enough from disk can cause disk IO to spike in a way that makes all other processes wait for available disk IO?
It's not a single process.
Thunderbird's newest defaults are to syncronize every folder and it will open one connectrion per folder in order to do that.
This is what I suspected, but failed to verify.
I really need to look into TBird's latest settings.
Appropriate disk/swappiness/readahead/cache tuning is required - along with enough ram to cope with several users trying this at once.
The underlaying filesystem is also important. as a for-instance, this kind of load with maildir format will effectively _kill_ a GFS box.
Our fix was to install systemwide defaults on managed systems to not synchronize (not needed, the home directories are on network storage anyway) and to ask people to check/unset this on laptops/etc
Thanks for your insights, Alan.
-- -Eric 'shubes'
On 11/04/2011 11:58, Alan Brown wrote:
From: Timo Sirainen <tss@iki.fi> Subject: Re: [Dovecot] IO rate quotas?
So a single process that is reading files fast enough from disk can cause disk IO to spike in a way that makes all other processes wait for available disk IO?
It's not a single process.
Thunderbird's newest defaults are to syncronize every folder and it will open one connectrion per folder in order to do that.
Appropriate disk/swappiness/readahead/cache tuning is required - along with enough ram to cope with several users trying this at once.
I don't really know what I'm talking about, but newer kernels have pretty strong support for io quotas, eg:
http://kernelnewbies.org/Linux_2_6_37#head-7f5782ae3ce80a383737e2e1bbc3b9ac9...
It would require some modifications to dovecot to apply cgroups, but it seems like a nice feature and I doubt that it would require too much sponsorship to implement? (Timo?)
Failing that you can apply ionice to a certain extent?
I doubt you can do much with iptables firewalling since it can't see inside the imap stream - perhaps an incoming max packet/ip restriction would be a rough approximation... Can't see it being effective though?
I'm a little surprised that a small number of users can take down your (1and1?) box? I think it's worth looking at your bottleneck a little more - I have a small server and I can't really do much to it despite a 40mbit broadband connection at this end?
As an aside, whilst cheap German/French hosting with 1and1 (or Hetzner, et al) looks good, I jumped and bought my own rack space some time back. A 1/4 rack with decent bandwidth only costs me the price of a reasonable 1and1 server, despite the rack being in the relatively more expensive UK... It's nice having the rack within a short drive and the incremental cost of a pair of supermicro servers (for redundancy, vs just a single server) is really no big beans. Give it some thought as a longer term plan?
Good luck
Ed W
participants (5)
-
Alan Brown
-
Ed W
-
Eric Shubert
-
Kádár Tamás (KTamas)
-
Stan Hoeppner