[Dovecot] enourmous amount of disc writes
Hi List, on my mailserver (exim4, dovecot 1.0.0, postgresql, ubuntu dapper, about 500 users) I record an enourmous amount of disc writes (up to 18000 blocks written each second) in the time where most of users are active. This IO activity causes wait-states on the disc interface (RAID5) and finally slows down my mailmachine. Is there a explanation for these disc writes respectively is there a possibility to minimize this activity. Stopping Dovecot on this machine causes stopping most of the disc activity.
Regards Rainer Sigl
--
Rainer Sigl, | email: sigl@mpe.mpg.de MPI f. extraterrestrische Physik, | phone: +49 (89) 30000-3557 Postfach 1312, | fax: +49 (89) 30000-3569 D-85741 Garching, Germany | web: http://www.mpe.mpg.de/~sigl/
Rainer Sigl schrieb:
Hi List, on my mailserver (exim4, dovecot 1.0.0, postgresql, ubuntu dapper, about 500 users) I record an enourmous amount of disc writes (up to 18000 blocks written each second) in the time where most of users are active. This IO activity causes wait-states on the disc interface (RAID5) and finally slows down my mailmachine. Is there a explanation for these disc writes respectively is there a possibility to minimize this activity. Stopping Dovecot on this machine causes stopping most of the disc activity.
Regards Rainer Sigl
Hi Rainer first you should upgrade to the latest stable of the 1.x branch or better 1.1.x, also dapper is very old perhaps try hardy
then retest, i run 2000 users on suse with dovecot 1.0.15 and mysql on sata 3ware hardware raid 1 500 GB on ext3 without any problems
-- Best Regards
MfG Robert Schetterer
Germany/Munich/Bavaria
Hello,
You might also want to enable "noatime" in the mount options for the volume housing the data... if you're using maildir, this *could* reduce the number of writes by half, theoretically!
Richard.
Robert Schetterer wrote:
Rainer Sigl schrieb:
Hi List, on my mailserver (exim4, dovecot 1.0.0, postgresql, ubuntu dapper, about 500 users) I record an enourmous amount of disc writes (up to 18000 blocks written each second) in the time where most of users are active. This IO activity causes wait-states on the disc interface (RAID5) and finally slows down my mailmachine. Is there a explanation for these disc writes respectively is there a possibility to minimize this activity. Stopping Dovecot on this machine causes stopping most of the disc activity.
Regards Rainer Sigl
Hi Rainer first you should upgrade to the latest stable of the 1.x branch or better 1.1.x, also dapper is very old perhaps try hardy
then retest, i run 2000 users on suse with dovecot 1.0.15 and mysql on sata 3ware hardware raid 1 500 GB on ext3 without any problems
-- Richard Hobbs (IT Specialist) Toshiba Research Europe Ltd. - Cambridge Research Laboratory Email: richard.hobbs@crl.toshiba.co.uk Web: http://www.toshiba-europe.com/research/ Tel: +44 1223 436999 Mobile: +44 7811 803377
Hi Richard, many thanks for you answer. yes, this solution did we discuss as well. For a future testing system I will try this. But I can't use it on the living system in the moment. Regards Rainer
Richard Hobbs wrote:
Hello,
You might also want to enable "noatime" in the mount options for the volume housing the data... if you're using maildir, this *could* reduce the number of writes by half, theoretically!
Richard.
Robert Schetterer wrote:
Rainer Sigl schrieb:
Hi List, on my mailserver (exim4, dovecot 1.0.0, postgresql, ubuntu dapper, about 500 users) I record an enourmous amount of disc writes (up to 18000 blocks written each second) in the time where most of users are active. This IO activity causes wait-states on the disc interface (RAID5) and finally slows down my mailmachine. Is there a explanation for these disc writes respectively is there a possibility to minimize this activity. Stopping Dovecot on this machine causes stopping most of the disc activity.
Regards Rainer Sigl
Hi Rainer first you should upgrade to the latest stable of the 1.x branch or better 1.1.x, also dapper is very old perhaps try hardy
then retest, i run 2000 users on suse with dovecot 1.0.15 and mysql on sata 3ware hardware raid 1 500 GB on ext3 without any problems
--
Rainer Sigl, | email: sigl@mpe.mpg.de MPI f. extraterrestrische Physik, | phone: +49 (89) 30000-3557 Postfach 1312, | fax: +49 (89) 30000-3569 D-85741 Garching, Germany | web: http://www.mpe.mpg.de/~sigl/
Dunno how much this applys to a local drive, but:
I run dovecot for about 40k users over nfs. Default install of redhat el5
The version of dovecot shipped with redhat caused all kinds of user
issues, so swapped it out with a pre 1.0 release of dovecot that I
knew was good.
About a year ago, I upgraded it to dovecot 1.1.3, and I freaked out,
cause we have no load on the mail/nfs servers at all. The nfs load
dropped to only 6% of what it was doing. No user complaints,
everything has been good for a year.
I was completely paranoid for a few days after the upgrade though,
cause my stats dropped so much.
Quoting Rainer Sigl sigl@mpe.mpg.de:
Hi Richard, many thanks for you answer. yes, this solution did we discuss as well. For a future testing system I will try this. But I can't use it on the living system in the moment. Regards Rainer
Richard Hobbs wrote:
Hello,
You might also want to enable "noatime" in the mount options for the volume housing the data... if you're using maildir, this *could* reduce the number of writes by half, theoretically!
Richard.
Robert Schetterer wrote:
Rainer Sigl schrieb:
Hi List, on my mailserver (exim4, dovecot 1.0.0, postgresql, ubuntu dapper, about 500 users) I record an enourmous amount of disc writes (up to 18000 blocks written each second) in the time where most of users are active. This IO activity causes wait-states on the disc interface (RAID5) and finally slows down my mailmachine. Is there a explanation for these disc writes respectively is there a possibility to minimize this activity. Stopping Dovecot on this machine causes stopping most of the disc
activity.Regards Rainer Sigl
Hi Rainer first you should upgrade to the latest stable of the 1.x branch or better 1.1.x, also dapper is very old perhaps try hardy
then retest, i run 2000 users on suse with dovecot 1.0.15 and mysql on sata 3ware hardware raid 1 500 GB on ext3 without any problems
--
Rainer Sigl, | email: sigl@mpe.mpg.de MPI f. extraterrestrische Physik, | phone: +49 (89) 30000-3557 Postfach 1312, | fax: +49 (89) 30000-3569 D-85741 Garching, Germany | web: http://www.mpe.mpg.de/~sigl/
Hi Robert, many thanks for your answer! Yes I'm aware of the necessity to upgrade the complete system. Beside this I wanted to search for a quick solution. Upgrading the system entails many other things and is in my eyes not possible on the living system. Regards Rainer
Robert Schetterer wrote:
Rainer Sigl schrieb:
Hi List, on my mailserver (exim4, dovecot 1.0.0, postgresql, ubuntu dapper, about 500 users) I record an enourmous amount of disc writes (up to 18000 blocks written each second) in the time where most of users are active. This IO activity causes wait-states on the disc interface (RAID5) and finally slows down my mailmachine. Is there a explanation for these disc writes respectively is there a possibility to minimize this activity. Stopping Dovecot on this machine causes stopping most of the disc activity.
Regards Rainer Sigl
Hi Rainer first you should upgrade to the latest stable of the 1.x branch or better 1.1.x, also dapper is very old perhaps try hardy
then retest, i run 2000 users on suse with dovecot 1.0.15 and mysql on sata 3ware hardware raid 1 500 GB on ext3 without any problems
--
Rainer Sigl, | email: sigl@mpe.mpg.de MPI f. extraterrestrische Physik, | phone: +49 (89) 30000-3557 Postfach 1312, | fax: +49 (89) 30000-3569 D-85741 Garching, Germany | web: http://www.mpe.mpg.de/~sigl/
Hi Robert, many thanks for your answer! Yes I'm aware of the necessity to upgrade the complete system. Beside this I wanted to search for a quick solution. Upgrading the system entails many other things and is in my eyes not possible on the living system. Did you ever measure the ioactiviy with iostat on your machine? Please tell me the output values. Which hardware do you use? I have a RAID5 with 3ware 7000 series. Regards Rainer
Robert Schetterer wrote:
Rainer Sigl schrieb:
Hi List, on my mailserver (exim4, dovecot 1.0.0, postgresql, ubuntu dapper, about 500 users) I record an enourmous amount of disc writes (up to 18000 blocks written each second) in the time where most of users are active. This IO activity causes wait-states on the disc interface (RAID5) and finally slows down my mailmachine. Is there a explanation for these disc writes respectively is there a possibility to minimize this activity. Stopping Dovecot on this machine causes stopping most of the disc activity.
Regards Rainer Sigl
Hi Rainer first you should upgrade to the latest stable of the 1.x branch or better 1.1.x, also dapper is very old perhaps try hardy
then retest, i run 2000 users on suse with dovecot 1.0.15 and mysql on sata 3ware hardware raid 1 500 GB on ext3 without any problems
--
Rainer Sigl, | email: sigl@mpe.mpg.de MPI f. extraterrestrische Physik, | phone: +49 (89) 30000-3557 Postfach 1312, | fax: +49 (89) 30000-3569 D-85741 Garching, Germany | web: http://www.mpe.mpg.de/~sigl/
I think I mentioned this twice with little response by the DC community, but here goes a third time. I have things set up so that I can kill DC, swap in the new executables and restart DC, all in about 4 minutes or less. And if things go wrong, I can fall back just as fast. Users don't know that I've done anything (if the upgrade is good); I usually do it during lunchtime....so there shouldn't be a problem with doing it prime shift. So upgrade shouldn't be a big deal. Of course, I'm running a pretty vanilla DC install with no real plugin, no deliver, etc.
==== Once upon a time, the Internet was a friendly, neighbors-helping-neighbors small town, and no one locked their doors. Now it's like an apartment in Bed-Stuy: you need three heavy duty pick-proof locks, one of those braces that goes from the lock to the floor, and bars on the windows.... ==== Stewart Dean, Unix System Admin, Bard College, New York 12504 sdean@bard.edu voice: 845-758-7475, fax: 845-758-7035
Stewart Dean wrote:
I think I mentioned this twice with little response by the DC community, but here goes a third time. I have things set up so that I can kill DC, swap in the new executables and restart DC, all in about 4 minutes or less. And if things go wrong, I can fall back just as fast. Users don't know that I've done anything (if the upgrade is good); I usually do it during lunchtime....so there shouldn't be a problem with doing it prime shift. So upgrade shouldn't be a big deal. Of course, I'm running a pretty vanilla DC install with no real plugin, no deliver, etc.
I use linux-vservers.
If you really want you can take a copy of the whole vserver, upgrade the copy, I map in the storage as a bind mount (but nfs, etc would work) and then you can test your new server.
When it works snapshot the old machine to be sure, upgrade the new machine and off you go.
Also vservers reboot really quick so if you ever get nervous that somthing is working right you have the option to bounce the whole vserver and make sure everything is fresh and it takes perhaps only a few seconds.
Also vservers are just a fancy way to do chroot's, so of course you can also strip your server down into lots of sandboxes, each running a single service, so bouncing one service could have very little impact on the rest of the whole installation
Also makes boosting the setup across to a new physical machine very easily also
Good luck
Ed W
Rainer Sigl schrieb:
Hi Robert, many thanks for your answer! Yes I'm aware of the necessity to upgrade the complete system. Beside this I wanted to search for a quick solution. Upgrading the system entails many other things and is in my eyes not possible on the living system.
look for parameters in dovecot.conf example which might help you with your problem
Did you ever measure the ioactiviy with iostat on your machine?
no i havent, why should i ,its running fine since years, i have a meassure from an nearly equal server which runs sharedweb apache2, it shows nearly null iostat, only during a rsync back there are a few outbreaks but that might be not comparable to a imap server
Please tell me the output values. Which hardware do you use? I have a RAID5 with 3ware 7000 series.
i have 3ware 9550SX SATA-RAID
Regards Rainer
Robert Schetterer wrote:
Rainer Sigl schrieb:
Hi List, on my mailserver (exim4, dovecot 1.0.0, postgresql, ubuntu dapper, about 500 users) I record an enourmous amount of disc writes (up to 18000 blocks written each second) in the time where most of users are active. This IO activity causes wait-states on the disc interface (RAID5) and finally slows down my mailmachine. Is there a explanation for these disc writes respectively is there a possibility to minimize this activity. Stopping Dovecot on this machine causes stopping most of the disc activity.
Regards Rainer Sigl
Hi Rainer first you should upgrade to the latest stable of the 1.x branch or better 1.1.x, also dapper is very old perhaps try hardy
then retest, i run 2000 users on suse with dovecot 1.0.15 and mysql on sata 3ware hardware raid 1 500 GB on ext3 without any problems
-- Best Regards
MfG Robert Schetterer
Germany/Munich/Bavaria
participants (6)
-
Ed W
-
Patrick Domack
-
Rainer Sigl
-
Richard Hobbs
-
Robert Schetterer
-
Stewart Dean