[Dovecot] Benchmarking

Thu Jan 25 20:25:03 UTC 2007

I wasted too much time today doing this, but they do show a few
interesting things:

1) mmap_disable=yes is faster than mmap_disable=no. I didn't expect
this. I'm not yet sure why this is, but possibly because then it updates
the mail index by reading the changes from dovecot.index.log, instead of
reopening and re-mmaping dovecot.index. At least that's the biggest
difference between them.

2) Dovecot is faster than Cyrus, at least in test. I didn't really
expect this either, because with maildir Dovecot has to do extra
rename() calls for all flag changes. I'm guessing it's expunges that are
slowing Cyrus down the most (that causes it to rewrite its cache file).

3) fsync_disable=yes + mmap_disable=yes almost doubles the performance
with 1 client, and over quadruples the performance with 10 clients. Why
fsync_disable=yes + mmap_disable=no doesn't give that big of a
performance difference? I've no idea. This is anyway such a large
performance increment that I'll probably go and backport this setting to
v1.0 too :)

4) Dovecot's mbox code kicks maildir's ass.

5) dbox is surprisingly slow. I'll have to find out why. I don't think
it should be that slow.

6) Courier.. Well, it probably doesn't come as a shock that Dovecot is
almost twice as fast. Note that Courier doesn't call fsync(), so it
can't be compared to Dovecot's fsync_disable=no numbers or to Cyrus. Oh,
one other difference between Courier and others was that it didn't
support LITERAL+ extension, so it maybe could have done a bit less
appends than others.

7) CVS HEAD and branch_1_0 don't seem to have that big of a performance
difference.

All the tests were done using my IMAP tester:
http://dovecot.org/tools/imaptest.c

The mailbox data that was being appended was Dovecot's mailing list:
http://dovecot.org/archives/dovecot.mbox

The linefeeds were converted to CR+LF. However, for the Dovecot CVS HEAD
runs the mails contained also those extra X-UID, Status, etc. headers
which I had to remove before Cyrus would accept them (the lines didn't
contain CRs). So CVS HEAD was actually doing a bit more work than the
others.

System:

Intel Core 2 Duo E6600, 2GB memory, 300GB Seagate ATA disk with XFS
Linux 2.6.19.2, x86-64 SMP, Debian unstable

Dovecot: CVS HEAD unless specified otherwise
Cyrus: v2.2.13-10 (Debian) - NOTE: comparable to fsync_disable:no
Courier: v4.1.1.20060828-2 (Debian) - NOTE: comparable to fsync_disable:yes!

Settings:

Dovecot: login_process_per_connection=no, login_processes_count=10,
  mail_drop_priv_before_exec=yes, passdb+userdb = passwd-file
Cyrus: Default Debian settings, except sasl_pwcheck_method: alwaystrue
Courier: Default Debian settings, using PAM + passwd

Results:

imaptest parameters: clients=1 msgs=30 secs=120 seed=1

mmap_disable:no fsync_disable:no
Logi Sele Fetc Fet2 Stor Dele Expu Appe Disc 
1135 1135 1134 1134  577  298  298  378 1134
1125 1125 1124 1124  573  295  295  373 1124
1123 1123 1122 1122  572  294  294  371 1122

mmap_disable:yes fsync_disable:no
Logi Sele Fetc Fet2 Stor Dele Expu Appe Disc 
1401 1401 1400 1400  710  388  388  480 1400
1398 1398 1397 1397  710  386  385  478 1397
1395 1395 1394 1394  709  384  384  476 1394

mmap_disable:no fsync_disable:yes
Logi Sele Fetc Fet2 Stor Dele Expu Appe Disc 
1356 1356 1355 1355  687  369  369  460 1355
1360 1360 1359 1359  690  371  370  462 1359
1363 1363 1362 1362  693  372  372  463 1362

mmap_disable:yes fsync_disable:yes
Logi Sele Fetc Fet2 Stor Dele Expu Appe Disc 
2486 2486 2485 2485 1221  669  669  811 2485
2475 2475 2474 2474 1216  667  667  807 2474
2497 2497 2496 2496 1226  673  673  815 2496

mmap_disable:yes fsync_disable:no - VERSION: CVS branch_1_0
Logi Sele Fetc Fet2 Stor Dele Expu Appe Disc 
1359 1359 1358 1358  690  370  370  461 1358 

mmap_disable:yes fsync_disable:no format:mbox
Logi Sele Fetc Fet2 Stor Dele Expu Appe Disc 
2101 2101 2100 2100 1046  568  568  684 2100 
2064 2064 2063 2063 1031  553  552  669 2063 

mmap_disable:yes fsync_disable:yes format:mbox
Logi Sele Fetc Fet2 Stor Dele Expu Appe Disc 
3362 3362 3361 3361 1653  912  911 1095 3361 

mmap_disable:yes fsync_disable:yes format:dbox
Logi Sele Fetc Fet2 Stor Dele Expu Appe Disc 
 820  820  819  819  406  209  209  267  819 

Cyrus:
Logi Sele Fetc Fet2 Stor Dele Expu Appe Disc 
 535  535  533  533  261  148  148  195  534 
 520  520  519  519  254  142  142  189  520 

Courier:
Logi Sele Fetc Fet2 Stor Dele Expu Appe Disc 
 833  833  832  831  407  250  250  350  832 
 822  822  821  820  404  245  245  344  821 

---

imaptest parameters: clients=1 msgs=1000 secs=120 seed=1

mmap_disable:no fsync_disable:no
Logi Sele Fetc Fet2 Stor Dele Expu Appe Disc 
 365  365  364  364  181  226  226  487  364
 376  376  375  375  186  236  236  503  375
 369  369  368  368  183  230  230  493  368

mmap_disable:yes fsync_disable:no
Logi Sele Fetc Fet2 Stor Dele Expu Appe Disc 
 426  426  425  425  212  269  269  564  425
 428  428  427  427  213  271  271  567  427
 424  424  423  423  211  267  266  562  423

mmap_disable:no fsync_disable:yes
Logi Sele Fetc Fet2 Stor Dele Expu Appe Disc 
 497  497  496  496  240  320  320  664  496
 501  501  500  500  243  322  322  671  500
 495  495  494  494  240  318  317  662  494

mmap_disable:yes fsync_disable:yes
Logi Sele Fetc Fet2 Stor Dele Expu Appe Disc 
 742  742  741  741  378  500  500  991  741
 747  747  746  746  379  504  504  996  746
 743  743  742  742  378  501  501  992  742

mmap_disable:yes fsync_disable:no format:mbox
Logi Sele Fetc Fet2 Stor Dele Expu Appe Disc 
 485  485  484  484  235  311  311  649  484 

Cyrus:
Logi Sele Fetc Fet2 Stor Dele Expu Appe Disc 
 290  290  289  289  144  173  173  394  289 
 286  286  285  285  141  172  172  387  285 

Courier:
Logi Sele Fetc Fet2 Stor Dele Expu Appe Disc 
 461  461  460  459  223  295  295  615  460 
 458  458  457  457  222  293  293  611  457 

---

imaptest parameters: clients=10 msgs=30 secs=120 seed=1

mmap_disable:no fsync_disable:no
Logi Sele Fetc Fet2 Stor Dele Expu Appe Disc
 444  442  432  430  210  214  213  490  434
 479  479  470  468  213  216  214  508  469
 476  472  464  463  211  206  203  485  460

mmap_disable:yes fsync_disable:no
Logi Sele Fetc Fet2 Stor Dele Expu Appe Disc
 440  440  433  431  195  184  183  460  430
 436  436  430  430  199  177  176  455  426
 435  432  425  425  208  165  165  455  425

mmap_disable:no fsync_disable:yes
Logi Sele Fetc Fet2 Stor Dele Expu Appe Disc
 680  679  674  670  325  299  298  721  670
 676  675  669  669  307  297  297  693  666
 688  687  677  677  311  305  303  731  678

mmap_disable:yes fsync_disable:yes
Logi Sele Fetc Fet2 Stor Dele Expu Appe Disc 
1804 1803 1793 1790  904  765  764 1901 1792 
1804 1804 1801 1800  888  789  787 1949 1793 
1800 1800 1789 1786  875  799  797 1964 1782 

mmap_disable:yes fsync_disable:no format:mbox
Logi Sele Fetc Fet2 Stor Dele Expu Appe Disc 
 644  644  633  633  305  224  221  526  634 

Cyrus:
Logi Sele Fetc Fet2 Stor Dele Expu Appe Disc 
 444  441  429  421  209  186  184  381  436 
 424  422  420  412  193  186  185  395  415 
 443  442  431  419  227  180  180  373  433 

Courier:
Logi Sele Fetc Fet2 Stor Dele Expu Appe Disc 
 981  976  974  973  463  406  405  667  971 
 971  966  965  965  451  384  383  655  961 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url : http://dovecot.org/pipermail/dovecot/attachments/20070125/e38618a6/attachment.pgp 