I wasted too much time today doing this, but they do show a few interesting things:
mmap_disable=yes is faster than mmap_disable=no. I didn't expect this. I'm not yet sure why this is, but possibly because then it updates the mail index by reading the changes from dovecot.index.log, instead of reopening and re-mmaping dovecot.index. At least that's the biggest difference between them.
Dovecot is faster than Cyrus, at least in test. I didn't really expect this either, because with maildir Dovecot has to do extra rename() calls for all flag changes. I'm guessing it's expunges that are slowing Cyrus down the most (that causes it to rewrite its cache file).
fsync_disable=yes + mmap_disable=yes almost doubles the performance with 1 client, and over quadruples the performance with 10 clients. Why fsync_disable=yes + mmap_disable=no doesn't give that big of a performance difference? I've no idea. This is anyway such a large performance increment that I'll probably go and backport this setting to v1.0 too :)
Dovecot's mbox code kicks maildir's ass.
dbox is surprisingly slow. I'll have to find out why. I don't think it should be that slow.
Courier.. Well, it probably doesn't come as a shock that Dovecot is almost twice as fast. Note that Courier doesn't call fsync(), so it can't be compared to Dovecot's fsync_disable=no numbers or to Cyrus. Oh, one other difference between Courier and others was that it didn't support LITERAL+ extension, so it maybe could have done a bit less appends than others.
CVS HEAD and branch_1_0 don't seem to have that big of a performance difference.
All the tests were done using my IMAP tester: http://dovecot.org/tools/imaptest.c
The mailbox data that was being appended was Dovecot's mailing list: http://dovecot.org/archives/dovecot.mbox
The linefeeds were converted to CR+LF. However, for the Dovecot CVS HEAD runs the mails contained also those extra X-UID, Status, etc. headers which I had to remove before Cyrus would accept them (the lines didn't contain CRs). So CVS HEAD was actually doing a bit more work than the others.
System:
Intel Core 2 Duo E6600, 2GB memory, 300GB Seagate ATA disk with XFS Linux 2.6.19.2, x86-64 SMP, Debian unstable
Dovecot: CVS HEAD unless specified otherwise Cyrus: v2.2.13-10 (Debian) - NOTE: comparable to fsync_disable:no Courier: v4.1.1.20060828-2 (Debian) - NOTE: comparable to fsync_disable:yes!
Settings:
Dovecot: login_process_per_connection=no, login_processes_count=10, mail_drop_priv_before_exec=yes, passdb+userdb = passwd-file Cyrus: Default Debian settings, except sasl_pwcheck_method: alwaystrue Courier: Default Debian settings, using PAM + passwd
Results:
imaptest parameters: clients=1 msgs=30 secs=120 seed=1
mmap_disable:no fsync_disable:no Logi Sele Fetc Fet2 Stor Dele Expu Appe Disc 1135 1135 1134 1134 577 298 298 378 1134 1125 1125 1124 1124 573 295 295 373 1124 1123 1123 1122 1122 572 294 294 371 1122
mmap_disable:yes fsync_disable:no Logi Sele Fetc Fet2 Stor Dele Expu Appe Disc 1401 1401 1400 1400 710 388 388 480 1400 1398 1398 1397 1397 710 386 385 478 1397 1395 1395 1394 1394 709 384 384 476 1394
mmap_disable:no fsync_disable:yes Logi Sele Fetc Fet2 Stor Dele Expu Appe Disc 1356 1356 1355 1355 687 369 369 460 1355 1360 1360 1359 1359 690 371 370 462 1359 1363 1363 1362 1362 693 372 372 463 1362
mmap_disable:yes fsync_disable:yes Logi Sele Fetc Fet2 Stor Dele Expu Appe Disc 2486 2486 2485 2485 1221 669 669 811 2485 2475 2475 2474 2474 1216 667 667 807 2474 2497 2497 2496 2496 1226 673 673 815 2496
mmap_disable:yes fsync_disable:no - VERSION: CVS branch_1_0 Logi Sele Fetc Fet2 Stor Dele Expu Appe Disc 1359 1359 1358 1358 690 370 370 461 1358
mmap_disable:yes fsync_disable:no format:mbox Logi Sele Fetc Fet2 Stor Dele Expu Appe Disc 2101 2101 2100 2100 1046 568 568 684 2100 2064 2064 2063 2063 1031 553 552 669 2063
mmap_disable:yes fsync_disable:yes format:mbox Logi Sele Fetc Fet2 Stor Dele Expu Appe Disc 3362 3362 3361 3361 1653 912 911 1095 3361
mmap_disable:yes fsync_disable:yes format:dbox Logi Sele Fetc Fet2 Stor Dele Expu Appe Disc 820 820 819 819 406 209 209 267 819
Cyrus: Logi Sele Fetc Fet2 Stor Dele Expu Appe Disc 535 535 533 533 261 148 148 195 534 520 520 519 519 254 142 142 189 520
Courier: Logi Sele Fetc Fet2 Stor Dele Expu Appe Disc 833 833 832 831 407 250 250 350 832 822 822 821 820 404 245 245 344 821
imaptest parameters: clients=1 msgs=1000 secs=120 seed=1
mmap_disable:no fsync_disable:no Logi Sele Fetc Fet2 Stor Dele Expu Appe Disc 365 365 364 364 181 226 226 487 364 376 376 375 375 186 236 236 503 375 369 369 368 368 183 230 230 493 368
mmap_disable:yes fsync_disable:no Logi Sele Fetc Fet2 Stor Dele Expu Appe Disc 426 426 425 425 212 269 269 564 425 428 428 427 427 213 271 271 567 427 424 424 423 423 211 267 266 562 423
mmap_disable:no fsync_disable:yes Logi Sele Fetc Fet2 Stor Dele Expu Appe Disc 497 497 496 496 240 320 320 664 496 501 501 500 500 243 322 322 671 500 495 495 494 494 240 318 317 662 494
mmap_disable:yes fsync_disable:yes Logi Sele Fetc Fet2 Stor Dele Expu Appe Disc 742 742 741 741 378 500 500 991 741 747 747 746 746 379 504 504 996 746 743 743 742 742 378 501 501 992 742
mmap_disable:yes fsync_disable:no format:mbox Logi Sele Fetc Fet2 Stor Dele Expu Appe Disc 485 485 484 484 235 311 311 649 484
Cyrus: Logi Sele Fetc Fet2 Stor Dele Expu Appe Disc 290 290 289 289 144 173 173 394 289 286 286 285 285 141 172 172 387 285
Courier: Logi Sele Fetc Fet2 Stor Dele Expu Appe Disc 461 461 460 459 223 295 295 615 460 458 458 457 457 222 293 293 611 457
imaptest parameters: clients=10 msgs=30 secs=120 seed=1
mmap_disable:no fsync_disable:no Logi Sele Fetc Fet2 Stor Dele Expu Appe Disc 444 442 432 430 210 214 213 490 434 479 479 470 468 213 216 214 508 469 476 472 464 463 211 206 203 485 460
mmap_disable:yes fsync_disable:no Logi Sele Fetc Fet2 Stor Dele Expu Appe Disc 440 440 433 431 195 184 183 460 430 436 436 430 430 199 177 176 455 426 435 432 425 425 208 165 165 455 425
mmap_disable:no fsync_disable:yes Logi Sele Fetc Fet2 Stor Dele Expu Appe Disc 680 679 674 670 325 299 298 721 670 676 675 669 669 307 297 297 693 666 688 687 677 677 311 305 303 731 678
mmap_disable:yes fsync_disable:yes Logi Sele Fetc Fet2 Stor Dele Expu Appe Disc 1804 1803 1793 1790 904 765 764 1901 1792 1804 1804 1801 1800 888 789 787 1949 1793 1800 1800 1789 1786 875 799 797 1964 1782
mmap_disable:yes fsync_disable:no format:mbox Logi Sele Fetc Fet2 Stor Dele Expu Appe Disc 644 644 633 633 305 224 221 526 634
Cyrus: Logi Sele Fetc Fet2 Stor Dele Expu Appe Disc 444 441 429 421 209 186 184 381 436 424 422 420 412 193 186 185 395 415 443 442 431 419 227 180 180 373 433
Courier: Logi Sele Fetc Fet2 Stor Dele Expu Appe Disc 981 976 974 973 463 406 405 667 971 971 966 965 965 451 384 383 655 961
On Thu, 2007-01-25 at 22:25 +0200, Timo Sirainen wrote:
Logi Sele Fetc Fet2 Stor Dele Expu Appe Disc 1135 1135 1134 1134 577 298 298 378 1134
Oh, and about these numbers. Number of logins (Logi) and number of appends (Appe) are the most important ones I think. All the numbers mean number of IMAP commands sent not the number of messages affected. For example "Expu" means number of EXPUNGE commands sent, not the actual number of messages expunged.
The benchmark does these commands:
LOGIN user pass SELECT INBOX FETCH 1:* (UID FLAGS ENVELOPE BODY) FETCH one-random-message (BODY[]) STORE random-messageset FLAGS.SILENT (random-flags and random-keywords-from-5-different) STORE random-messageset +FLAGS.SILENT \Deleted EXPUNGE APPEND new message, unless we're over threshold
- randomly APPEND more messages LOGOUT
So the msgs-parameter to imaptest means how many messages it tries to keep in the mailbox. The msgs=30 works pretty well, but msgs=1000 was never really reached in the 120 second tests. It was more like 100-150 messages where it was kept. I guess I should try the msgs=1000 with longer test runs.
Timo Sirainen wrote:
Dovecot's mbox code kicks maildir's ass.
dbox is surprisingly slow. I'll have to find out why. I don't think it should be that slow.
does this means you suggest to use mbox in stead of maildir? and dbox is far from usable?
-- Levente "Si vis pacem para bellum!"
On 25.1.2007, at 23.06, Farkas Levente wrote:
Timo Sirainen wrote:
Dovecot's mbox code kicks maildir's ass.
dbox is surprisingly slow. I'll have to find out why. I don't
think it should be that slow.does this means you suggest to use mbox in stead of maildir?
No. The performance may be better, but reliability is worse. Also
this test didn't even try to simulate a real world scenario, so it's
possible that maildir is faster in real world.
and dbox is far from usable?
I don't know if far is the right word, but I don't really want people
to use it yet.
Timo Sirainen wrote:
I wasted too much time today doing this, but they do show a few interesting things:
Hi Timo...
Thanks for this info!
A couple of (probably dumb) questions ...
- fsync_disable=yes + mmap_disable=yes almost doubles the performance with 1 client, and over quadruples the performance with 10 clients. Why fsync_disable=yes + mmap_disable=no doesn't give that big of a performance difference? I've no idea. This is anyway such a large performance increment that I'll probably go and backport this setting to v1.0 too :)
In other words, these will be the new default settings? And I guess you also now recommend using these settings in existing setups, unless there is a platform/fs specific reason not to?
- Dovecot's mbox code kicks maildir's ass.
Hmmm... trying to understand the significance of this. I always thought maildir was better=faster. I can still readily see how maildir is more reliable (one message getting corrupted means you only lose that one message), but again, I always thought it was faster too.
- dbox is surprisingly slow. I'll have to find out why. I don't think it should be that slow.
:( from what I read about dbox, it looks very promising. Well, since it is still very new, I guess it isn't surprising. I'm sure you'll figure out why and soon it'll be kicking mbox's ass.
--
Best regards,
Charles
On 25.1.2007, at 23.10, Charles Marcus wrote:
- fsync_disable=yes + mmap_disable=yes almost doubles the performance with 1 client, and over quadruples the performance with 10 clients. Why fsync_disable=yes + mmap_disable=no doesn't give that big of a performance difference? I've no idea. This is anyway such a large performance increment that I'll probably go and backport this setting to v1.0 too :)
In other words, these will be the new default settings? And I guess
you also now recommend using these settings in existing setups,
unless there is a platform/fs specific reason not to?
I'm not sure if I'll change default settings. Maybe I should set
mmap_disable=yes at least. But fsync_disable=yes means only that
Dovecot doesn't do any fsync() calls, which means that if the
computer crashes at that point, Dovecot could have lied to the client
that "OK, your saved message is safe" while in reality it isn't. So
it'll default to no, but if you don't care about such problems that
much you can enable it..
- Dovecot's mbox code kicks maildir's ass.
Hmmm... trying to understand the significance of this. I always
thought maildir was better=faster. I can still readily see how
maildir is more reliable (one message getting corrupted means you
only lose that one message), but again, I always thought it was
faster too.
I think most people think maildir is faster only because all other
mbox readers are so horribly slow. And that's because they don't use
index files. Pretty much the only thing that's slower in mbox is
expunging old messages, which causes a lot of data to be moved around
inside the mbox file.
Also maildir's reliability isn't just because of that "only one
message getting corrupted", but it's that once the message has been
written, there's no way for it to get corrupted unless the filesystem
breaks. With mbox the data is constantly being moved around and it
can break both because of software and because computer crashes while
moving the data around.
I'm sorry Timo, I must be dense... ;)
I'm not sure if I'll change default settings. Maybe I should set mmap_disable=yes at least. But fsync_disable=yes means only that Dovecot doesn't do any fsync() calls, which means that if the computer crashes at that point,
Computer, meaning the dovecot server? Or the Client?
Dovecot could have lied to the client that "OK, your saved message is safe" while in reality it isn't. So it'll default to no, but if you don't care about such problems that much you can enable it..
Sounds like you meant the server/dovecot, since if the Client crashed *before* the operation was complete,. it could not have gotten an "OK, your saved message is safe". No, I wouldn't be worried about that small of a chance - my server has never crashed (Charles runs to knock a hole in a 4x4 block of wood), so, I'd definitely be willing to risk that tiny chance of the server crashing at just the wrong time, to reap the speed benefits you described during normal operations.
I think most people think maildir is faster only because all other mbox readers are so horribly slow.
By reader, I'm assuming you mean server? Or do you actually mean that the mail Client can have such a huge impact on the speed when talking to an IMAP server that uses mbox on the backend?
Also maildir's reliability isn't just because of that "only one message getting corrupted", but it's that once the message has been written, there's no way for it to get corrupted unless the filesystem breaks. With mbox the data is constantly being moved around and it can break both because of software and because computer crashes while moving the data around.
Makes perfect sense. Thanks for taking the time to explain things that must seem painfully obvious to you... :)
--
Best regards,
Charles
On 25.1.2007, at 23.43, Charles Marcus wrote:
Dovecot could have lied to the client that "OK, your saved message is safe" while in reality it isn't. So it'll default to no, but if you don't care about such problems that much you can enable it..
Sounds like you meant the server/dovecot, since if the Client
crashed *before* the operation was complete,. it could not have
gotten an "OK, your saved message is safe".
Yep.
I think most people think maildir is faster only because all other
mbox readers are so horribly slow.By reader, I'm assuming you mean server? Or do you actually mean
that the mail Client can have such a huge impact on the speed when
talking to an IMAP server that uses mbox on the backend?
I mean anything that accesses mboxes directly: UW-IMAP, mutt, pine,
and so on. IMAP client doesn't have anything to do with it.
On Thu, Jan 25, 2007 at 10:25:03PM +0200, Timo Sirainen wrote:
- Dovecot's mbox code kicks maildir's ass.
Perhaps this is just a filesystem issue, XFS is known for handling well large streaming-like data and is less optimized for many small files. Maybe reiserfs would give you the opposite relationship.
Intel Core 2 Duo E6600, 2GB memory, 300GB Seagate ATA disk with XFS Linux 2.6.19.2, x86-64 SMP, Debian unstable
Axel.Thimm at ATrpms.net
On Fri, 2007-01-26 at 21:19 +0100, Axel Thimm wrote:
On Thu, Jan 25, 2007 at 10:25:03PM +0200, Timo Sirainen wrote:
- Dovecot's mbox code kicks maildir's ass.
Perhaps this is just a filesystem issue, XFS is known for handling well large streaming-like data and is less optimized for many small files. Maybe reiserfs would give you the opposite relationship.
Ah, yes, I didn't think there would be that big of a difference. Creating a 100MB file inside the XFS system and using it as reiserfs gave these numbers:
mmap_disable:yes fsync_disable:no format:maildir reiserfs Logi Sele Fetc Fet2 Stor Dele Expu Appe Disc 3864 3864 3863 3863 1883 1038 1038 1243 3863
mmap_disable:yes fsync_disable:no format:mbox reiserfs Logi Sele Fetc Fet2 Stor Dele Expu Appe Disc 2289 2289 2289 2289 1141 627 626 730 2288
With ext3:
mmap_disable:yes fsync_disable:no format:maildir ext2 Logi Sele Fetc Fet2 Stor Dele Expu Appe Disc 3987 3987 3986 3986 1934 1076 1076 1288 3986
So ext3 is even faster than reiserfs..
I suppose all those benchmarks should be run again with ext3.
Hi Timo, many thanks for the numbers.
Maybe having a raiserfs/etx3 on top of XFS have some perfomance implications. But it should work fine for general benchmark
It would be nice to have a benchmark using the same disk and the filesystem formated on a disk partition. I say the same disk because some part of a hard disk are faster than other (inner/outer tracks)
My current setup is ext3 and maildir, so I think I should be ok with the perfomance. ;)
Regards, Oliver
Timo Sirainen wrote:
Ah, yes, I didn't think there would be that big of a difference. Creating a 100MB file inside the XFS system and using it as reiserfs gave these numbers:
-- Oliver Schulze L. | Get my e-mail after a captcha in: Asuncion - Paraguay | http://tinymailto.com/oliver
* On 28/01/07 12:07 -0300, Oliver Schulze L. wrote:
| Hi Timo,
| many thanks for the numbers.
|
| Maybe having a raiserfs/etx3 on top of XFS have some perfomance
| implications. But it should work fine for general benchmark
|
| It would be nice to have a benchmark using the same disk and
| the filesystem formated on a disk partition.
| I say the same disk because some part of a hard disk are faster than
| other (inner/outer tracks)
|
| My current setup is ext3 and maildir, so I think I should be ok
| with the perfomance. ;)
|
| Regards,
| Oliver
|
| Timo Sirainen wrote:
Hi Timo,
Could you please benchmark on FreeBSD as well? You still have access to
my serve.
This Linux stuff doesn't ring a nice bell on my ears ;)
-Wash
http://www.netmeister.org/news/learn2quote.html
DISCLAIMER: See http://www.wananchi.com/bms/terms.php
--
+======================================================================+
|\ _,,,---,,_ | Odhiambo Washington
On Sun, 2007-01-28 at 18:20 +0300, Odhiambo WASHINGTON wrote:
Could you please benchmark on FreeBSD as well? You still have access to my serve. This Linux stuff doesn't ring a nice bell on my ears ;)
I'd have to do pretty much all the benchmarks there all over again to get results that could be directly compared. You can do that yourself too. :)
Or then again with about 100 EUR donation I could go and buy a new hard disk with one partition reserved for benchmarking and the rest for different OSes, so all results could be directly compared ;)
BTW. I found out that when I compiled Dovecot without --enable-debug, the imaptest started giving errors quite easily. With --enable-debug (that I've been using myself always) I could run it for hours without a single error. This means that I'll still have to get those "new" bugs fixed before v1.0 too.
Timo Sirainen wrote:
So ext3 is even faster than reiserfs..
I suppose all those benchmarks should be run again with ext3.
An optimization for ext3 is to make sure b-trees are enabled by formatting with -O dir_index
This improves performance for directories with many small files in them, like maildirs
Mick
participants (7)
-
Axel Thimm
-
Charles Marcus
-
Farkas Levente
-
Mick T
-
Odhiambo WASHINGTON
-
Oliver Schulze L.
-
Timo Sirainen