[Dovecot] Zimbra benchmarking

Timo Sirainen tss at iki.fi
Fri Nov 30 12:38:29 EET 2007


Now that I have a working kvm setup, I thought I'd finally try how
Zimbra works. This is mainly some microbenchmarking, so it may not have
much to do with actual performance in real life.

Setup:

 - 1GB memory given to kvm (from host's 2GB)
 - Intel Core 2 6600 (kvm uses only one CPU)
 - CentOS 5
 - 15GB qcow2 image on XFS filesystem
 - Zimbra 5.0 RC2 RHEL5 x86_64
 - Dovecot latest hg, dbox format

Both Zimbra and Dovecot were tested running on the same kvm image. All
data could be cached in memory, so this didn't really test disk I/O.

Dovecot was run with fsync_disable=no. Running with =yes would have made
its performance even better.

Zimbra features/bugs
--------------------

Zimbra's SEARCH command doesn't support substring searching as IMAP
requires:

1 search text performance
* SEARCH 136 183 227 231 232 233 245
1 OK SEARCH completed
2 search text erformance
* SEARCH
2 OK SEARCH completed
3 search text performanc
* SEARCH
3 OK SEARCH completed

It's possible to search "performanc" in a non-standard way though:

4 search text "performanc*"
* SEARCH 136 183 227 231 232 233 245
4 OK SEARCH completed

This means that the upcoming Dovecot v1.1 is probably the only IMAP
server that supports IMAP-compatible full text search indexes with
incremental updates. (Cyrus Squat requires rebuilding the whole index
from scratch just for adding one mail, which makes it kind of
impractical.)

Zimbra doesn't support SORT or THREAD extensions.

I had trouble benchmarking more than 5 simultaneous connections in one
mailbox. I'm not sure if this is a configurable setting somewhere, or if
there's just some bug. The error messages that my imaptest gave looked
like Zimbra was buggy, but I didn't look at them too closely.

STORE command also seemed to be buggy, giving lots of "STORE failed"
errors, even with just one connection.

Zimbra uses 500MB of memory just to start up, so it's not exactly for
hosting small installations.

Login+Logout
------------

Dovecot:

./imaptest - select=0 secs=10 seed=0 clients=1
Logi Logo 
100% 100% 
3439 6878 
./imaptest - select=0 secs=10 seed=0 clients=10
Logi Logo 
100% 100% 
4415 8830 

Zimbra:

./imaptest - select=0 secs=10 seed=0 clients=1
Logi Logo 
100% 100% 
18032 36064 
./imaptest - select=0 secs=10 seed=0 clients=10
Logi Logo 
100% 100% 
25519 51056 

Looks like Dovecot's imap process creation hurts it a lot compared to
Zimbra's thread creation. Luckily IMAP connections are usually
long-living, so this shouldn't matter much, except for webmails for
which you can use imapproxy in the middle.

LIST "" *
---------

Dovecot:

./imaptest - select=0 logout=0 list=100 secs=10 seed=0 clients=1
Logi List 
100% 100% 
   1 181411 
./imaptest - select=0 logout=0 list=100 secs=10 seed=0 clients=10
Logi List 
100% 100% 
  10 133451 

Zimbra:

./imaptest - select=0 logout=0 list=100 secs=10 seed=0 clients=1
Logi List 
100% 100% 
   1 1021 
./imaptest - select=0 logout=0 list=100 secs=10 seed=0 clients=10
Logi List 
100% 100% 
  10  972 

Yes, Dovecot's LIST is over 100 times faster.

STATUS INBOX (MESSAGES UNSEEN RECENT)
-------------------------------------

100 messages in INBOX 

dovecot:

./imaptest - logout=0 select=0 status=100 secs=10 seed=0 clients=1
Logi Stat 
100% 100% 
   1 191003 
./imaptest - logout=0 select=0 status=100 secs=10 seed=0 clients=10
Logi Stat 
100% 100% 
  10 171171 

Zimbra:

./imaptest - logout=0 select=0 status=100 secs=10 seed=0 clients=1
Logi Stat 
100% 100% 
   1 3697 
./imaptest - logout=0 select=0 status=100 secs=10 seed=0 clients=10
Logi Stat 
100% 100% 
  10 7009 

FETCH 1:* (UID FLAGS ENVELOPE INTERNALDATE BODYSTRUCTURE)
---------------------------------------------------------

100 messages in INBOX.

dovecot:
./imaptest - logout=0 select=100 fetch=100 secs=10 seed=0 clients=1
Logi Sele Fetc 
100% 100% 100% 
   1    1 2769 
./imaptest - logout=0 select=100 fetch=100 secs=10 seed=0 clients=5
Logi Sele Fetc 
100% 100% 100% 
   5    5 2902 

Zimbra:

./imaptest - logout=0 select=100 fetch=100 secs=10 seed=0 clients=1
Logi Sele Fetc 
100% 100% 100% 
   1    1  203 
./imaptest - logout=0 select=100 fetch=100 secs=10 seed=0 clients=5
Logi Sele Fetc 
100% 100% 100% 
   5    5  258 

FETCH ? (BODY[])
----------------

Randomly fetch message body from 100 messages

Dovecot:

./imaptest - logout=0 select=100 fetch2=100 secs=10 seed=0 clients=1
Logi Sele Fet2 
100% 100% 100% 
           30% 
   1    1 51958 
./imaptest - logout=0 select=100 fetch2=100 secs=10 seed=0 clients=5
Logi Sele Fet2 
100% 100% 100% 
           30% 
   5    5 141397 

Zimbra:

./imaptest - logout=0 select=100 fetch2=100 secs=10 seed=0 clients=1
Logi Sele Fet2 
100% 100% 100% 
           30% 
   1    1  410 
./imaptest - logout=0 select=100 fetch2=100 secs=10 seed=0 clients=5
Logi Sele Fet2 
100% 100% 100% 
           30% 
   5    5 2291 

STORE ? FLAGS.SILENT (random flags and keywords)
------------------------------------------------

Dovecot:

./imaptest - logout=0 select=100 store=100 secs=10 seed=0 clients=1
Logi Sele Stor 
100% 100% 100% 
   1    1 15236 
./imaptest - logout=0 select=100 store=100 secs=10 seed=0 clients=5
Logi Sele Stor 
100% 100% 100% 
   5    5 5533 

Zimbra:

./imaptest - logout=0 select=100 store=100 secs=10 seed=0 clients=1
Logi Sele Stor 
100% 100% 100% 
   1    1  308 
./imaptest - logout=0 select=100 store=100 secs=10 seed=0 clients=5
Logi Sele Stor 
100% 100% 100% 
   5    5  319 

Zimbra gave lots of errors:
Error: STORE failed: STORE failed

APPEND
------

Appends to empty mailbox:

Dovecot:
1 client, 5 seconds: 3959 mails
1 client, 5 seconds, fsync_disable=yes: 21889 mails

Zimbra:
1 clients, 5 seconds: 222 mails
5 clients, 5 seconds: 322 mails

Zimbra is apparently building full text search indexes while appending,
so this test doesn't mean much until I can test Dovecot's performance
with Squat indexing.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url : http://dovecot.org/pipermail/dovecot/attachments/20071130/0b2fd6cf/attachment.bin 


More information about the dovecot mailing list