[Dovecot] 1 week after full migration to 1.1v
Ehlo all,
I've been working on a migration of about 240K users from one server with qmail, courier imap, maildrop and ldap to postfix, dovecot, lda +sieve. Before the migration the server and the storage were always on high load.
Before migration:
Storage IOPs = 4600 Read average = 2800 Write average = 1800
Storage CPU (Raid5)
Crl0 = 50%
Crl1 = 50%
Server Load = 120 %user = 35 %system = 60 %iowait = 5 %idle = 0
After migration:
Storage IOPs = 3000 Read average = 1200 Write average = 1800
Storage CPU (Raid5)
Crl0 = 40%
Crl1 = 40%
Server Load = 25 %user = 10 %system = 27 %iowait = 40 %idle = 23
And now the server responds so quickly to any imap command.
Dovecot is now the default imap/pop server here, Thanks Timo.
[]s,
Raphael Costa
Hi Raphael,
Can you also share the figures on the average mailbox size and number of cocurrent users in peak hour before and after migration? What is your hardware configuration (CPU / MEMORY / DISK TYPE )?
- Joe
----- Original Message ----- From: "Raphael Bittencourt S. Costa" raphaelbscosta@gmail.com To: "dovecot" dovecot@dovecot.org Sent: Monday, June 23, 2008 10:36 PM Subject: [Dovecot] 1 week after full migration to 1.1v
Ehlo all,
I've been working on a migration of about 240K users from one server with qmail, courier imap, maildrop and ldap to postfix, dovecot, lda +sieve. Before the migration the server and the storage were always on high load.
Before migration:
Storage IOPs = 4600 Read average = 2800 Write average = 1800
Storage CPU (Raid5) Crl0 = 50% Crl1 = 50%
Server Load = 120 %user = 35 %system = 60 %iowait = 5 %idle = 0
After migration:
Storage IOPs = 3000 Read average = 1200 Write average = 1800
Storage CPU (Raid5) Crl0 = 40% Crl1 = 40%
Server Load = 25 %user = 10 %system = 27 %iowait = 40 %idle = 23
And now the server responds so quickly to any imap command.
Dovecot is now the default imap/pop server here, Thanks Timo.
[]s,
Raphael Costa
Joe,
On Mon, 2008-06-23 at 22:58 +0800, Joe Wong wrote:
Hi Raphael,
Can you also share the figures on the average mailbox size
This is very hard to know... Because it depends on the hosting package, now every new mail account is created with 1,5G.
and number of cocurrent users in peak hour before and after migration?
The concurrent users doesnt change after the migration. Its about 820 concurrent imap users and 400 concurrent pop3 users.
What is your hardware configuration (CPU / MEMORY / DISK TYPE )?
Server is a Dell PE 1950, dual xeon(5160), 4G, 2x146SAS and the storage is Hitachi AMS 500, with 41x146G Fiber 15K.
- Joe
----- Original Message ----- From: "Raphael Bittencourt S. Costa" raphaelbscosta@gmail.com To: "dovecot" dovecot@dovecot.org Sent: Monday, June 23, 2008 10:36 PM Subject: [Dovecot] 1 week after full migration to 1.1v
Ehlo all,
I've been working on a migration of about 240K users from one server with qmail, courier imap, maildrop and ldap to postfix, dovecot, lda +sieve. Before the migration the server and the storage were always on high load.
Before migration:
Storage IOPs = 4600 Read average = 2800 Write average = 1800
Storage CPU (Raid5) Crl0 = 50% Crl1 = 50%
Server Load = 120 %user = 35 %system = 60 %iowait = 5 %idle = 0
After migration:
Storage IOPs = 3000 Read average = 1200 Write average = 1800
Storage CPU (Raid5) Crl0 = 40% Crl1 = 40%
Server Load = 25 %user = 10 %system = 27 %iowait = 40 %idle = 23
And now the server responds so quickly to any imap command.
Dovecot is now the default imap/pop server here, Thanks Timo.
[]s,
Raphael Costa
Raphael Bittencourt S. Costa wrote:
Ehlo all,
I've been working on a migration of about 240K users from one server with qmail, courier imap, maildrop and ldap to postfix, dovecot, lda +sieve. Before the migration the server and the storage were always on high load.
It almost doesn't seem enough of a drop...! Have you benchmarked the various apps a little? Postfix syncs disks like crazy to prevent mail loss in case of disaster - can I assume you have given some thought to tuning your file system which handles the spool files?
Good luck
Ed W
On Tue, 2008-06-24 at 11:44 +0100, Ed W wrote:
Raphael Bittencourt S. Costa wrote:
Ehlo all,
I've been working on a migration of about 240K users from one server with qmail, courier imap, maildrop and ldap to postfix, dovecot, lda +sieve. Before the migration the server and the storage were always on high load.
It almost doesn't seem enough of a drop...!
Really? The load of the qmail server were always on 120, but if something happens, like suddenly increase of mail deliver the server load jumps to 800. With postfix+dovecot the highest load I got was 35.
Have you benchmarked the various apps a little?
No, only dovecotxCourier.
Postfix syncs disks like crazy to prevent mail loss in case of disaster
Postfix queue is on local disk, and iostat only shows 6% of utilization.
- can I assume you have given some thought to tuning your file system which handles the spool files?
The server uses xfs for maildir partition and so on, it uses noatime,nodiratime,nobarrier,logbufs=8.
Good luck
Ed W
Hi
Sounds like you know what you are doing!
- can I assume you have given some thought to tuning your file system which handles the spool files?
The server uses xfs for maildir partition and so on, it uses noatime,nodiratime,nobarrier,logbufs=8.
I guess you have read around a little on XFS + maildir? There was some discussion a while back that showed it could sometimes get a little slow without some tweaking - I think the logbugs param you used was mentioned as the solution? Since you are IO constrainted did you try some benchmarks on other filesystems?
Good luck
Ed W
On Tue, 2008-06-24 at 15:20 +0100, Ed W wrote:
Hi
Sounds like you know what you are doing!
No, sounds like I'm thinking what I'm doing. :-D I really need more information to get more from the structure.
- can I assume you have given some thought to tuning your file system which handles the spool files?
The server uses xfs for maildir partition and so on, it uses noatime,nodiratime,nobarrier,logbufs=8.
I guess you have read around a little on XFS + maildir? There was some discussion a while back that showed it could sometimes get a little slow without some tweaking - I think the logbugs param you used was mentioned as the solution?
That's right.
Since you are IO constrainted did you try some benchmarks on other filesystems?
No. I used reiserfs before and I had many problems with it. The guy that was in charge before me tried xfs and showed me lots of good things about xfs. So, I decided to try it too. And backup is a problem here, so an security filesystem is very important for not losing mail.
Good luck
Ed W
Raphael Bittencourt S. Costa wrote:
And backup is a problem here, so an security filesystem is very important for not losing mail.
I have been using rsync (rsnapback) for many of my backups, but it struggles when you point it at hundreds of thousands of files in a directory. I have seen a number of solutions involving running multiple independent rsyncs for each maildir to keep the file numbers reasonable, but it's a bit of a fiddle
More recently I was reading webmail.us / rackspace's notes on their backup strategy and they have presentation on their use of S3 which is kind of cool. Reading between the lines they use something like an MD5 hash of all their files and then copy the files up to remote storage (S3) based on the file hash name. They then use some kind of index to record who has which files and which maildirs they belong in. Seems fairly cheap to make incremental backups and robust to maildirs tendency to move thousands of files from new/ to cur/...
There is an implementation of a similar algorithm via "brackup" - it might be interesting for your needs? I plan to do some more investigation into it for backing up our maildirs http://search.cpan.org/~bradfitz/Brackup/
Regards
Ed W
participants (3)
-
Ed W
-
Joe Wong
-
Raphael Bittencourt S. Costa