[Dovecot] Zimbra benchmarking

[Farkas Levente]

hi,
nice test, but what would be also useful to compare cpu and memory
usage! with zimbra it's more important then anything else.
i'd like to see a comparison what kind of hardware (cpu and memory)
required for: 10, 100, 500, 1000, 2000, 5000 mailbox server. this's
where dovecot a big winner even for 10 mailbox.
but to be fair zimbra is much more than dovecot it's integrate many
things (ldap user and group management, samba domain controller, virus
scanner, outlook connector (!) not just for email but also for calendar
and address book, etc...) and to put together such a system with dovecot
it's more than hard and may be impossible.


Timo Sirainen wrote:
> Now that I have a working kvm setup, I thought I'd finally try how
> Zimbra works. This is mainly some microbenchmarking, so it may not have
> much to do with actual performance in real life.
> 
> Setup:
> 
>  - 1GB memory given to kvm (from host's 2GB)
>  - Intel Core 2 6600 (kvm uses only one CPU)
>  - CentOS 5
>  - 15GB qcow2 image on XFS filesystem
>  - Zimbra 5.0 RC2 RHEL5 x86_64
>  - Dovecot latest hg, dbox format
> 
> Both Zimbra and Dovecot were tested running on the same kvm image. All
> data could be cached in memory, so this didn't really test disk I/O.
> 
> Dovecot was run with fsync_disable=no. Running with =yes would have made
> its performance even better.
> 
> Zimbra features/bugs
> --------------------
> 
> Zimbra's SEARCH command doesn't support substring searching as IMAP
> requires:
> 
> 1 search text performance
> * SEARCH 136 183 227 231 232 233 245
> 1 OK SEARCH completed
> 2 search text erformance
> * SEARCH
> 2 OK SEARCH completed
> 3 search text performanc
> * SEARCH
> 3 OK SEARCH completed
> 
> It's possible to search "performanc" in a non-standard way though:
> 
> 4 search text "performanc*"
> * SEARCH 136 183 227 231 232 233 245
> 4 OK SEARCH completed
> 
> This means that the upcoming Dovecot v1.1 is probably the only IMAP
> server that supports IMAP-compatible full text search indexes with
> incremental updates. (Cyrus Squat requires rebuilding the whole index
> from scratch just for adding one mail, which makes it kind of
> impractical.)
> 
> Zimbra doesn't support SORT or THREAD extensions.
> 
> I had trouble benchmarking more than 5 simultaneous connections in one
> mailbox. I'm not sure if this is a configurable setting somewhere, or if
> there's just some bug. The error messages that my imaptest gave looked
> like Zimbra was buggy, but I didn't look at them too closely.
> 
> STORE command also seemed to be buggy, giving lots of "STORE failed"
> errors, even with just one connection.
> 
> Zimbra uses 500MB of memory just to start up, so it's not exactly for
> hosting small installations.
> 
> Login+Logout
> ------------
> 
> Dovecot:
> 
> ./imaptest - select=0 secs=10 seed=0 clients=1
> Logi Logo 
> 100% 100% 
> 3439 6878 
> ./imaptest - select=0 secs=10 seed=0 clients=10
> Logi Logo 
> 100% 100% 
> 4415 8830 
> 
> Zimbra:
> 
> ./imaptest - select=0 secs=10 seed=0 clients=1
> Logi Logo 
> 100% 100% 
> 18032 36064 
> ./imaptest - select=0 secs=10 seed=0 clients=10
> Logi Logo 
> 100% 100% 
> 25519 51056 
> 
> Looks like Dovecot's imap process creation hurts it a lot compared to
> Zimbra's thread creation. Luckily IMAP connections are usually
> long-living, so this shouldn't matter much, except for webmails for
> which you can use imapproxy in the middle.
> 
> LIST "" *
> ---------
> 
> Dovecot:
> 
> ./imaptest - select=0 logout=0 list=100 secs=10 seed=0 clients=1
> Logi List 
> 100% 100% 
>    1 181411 
> ./imaptest - select=0 logout=0 list=100 secs=10 seed=0 clients=10
> Logi List 
> 100% 100% 
>   10 133451 
> 
> Zimbra:
> 
> ./imaptest - select=0 logout=0 list=100 secs=10 seed=0 clients=1
> Logi List 
> 100% 100% 
>    1 1021 
> ./imaptest - select=0 logout=0 list=100 secs=10 seed=0 clients=10
> Logi List 
> 100% 100% 
>   10  972 
> 
> Yes, Dovecot's LIST is over 100 times faster.
> 
> STATUS INBOX (MESSAGES UNSEEN RECENT)
> -------------------------------------
> 
> 100 messages in INBOX 
> 
> dovecot:
> 
> ./imaptest - logout=0 select=0 status=100 secs=10 seed=0 clients=1
> Logi Stat 
> 100% 100% 
>    1 191003 
> ./imaptest - logout=0 select=0 status=100 secs=10 seed=0 clients=10
> Logi Stat 
> 100% 100% 
>   10 171171 
> 
> Zimbra:
> 
> ./imaptest - logout=0 select=0 status=100 secs=10 seed=0 clients=1
> Logi Stat 
> 100% 100% 
>    1 3697 
> ./imaptest - logout=0 select=0 status=100 secs=10 seed=0 clients=10
> Logi Stat 
> 100% 100% 
>   10 7009 
> 
> FETCH 1:* (UID FLAGS ENVELOPE INTERNALDATE BODYSTRUCTURE)
> ---------------------------------------------------------
> 
> 100 messages in INBOX.
> 
> dovecot:
> ./imaptest - logout=0 select=100 fetch=100 secs=10 seed=0 clients=1
> Logi Sele Fetc 
> 100% 100% 100% 
>    1    1 2769 
> ./imaptest - logout=0 select=100 fetch=100 secs=10 seed=0 clients=5
> Logi Sele Fetc 
> 100% 100% 100% 
>    5    5 2902 
> 
> Zimbra:
> 
> ./imaptest - logout=0 select=100 fetch=100 secs=10 seed=0 clients=1
> Logi Sele Fetc 
> 100% 100% 100% 
>    1    1  203 
> ./imaptest - logout=0 select=100 fetch=100 secs=10 seed=0 clients=5
> Logi Sele Fetc 
> 100% 100% 100% 
>    5    5  258 
> 
> FETCH ? (BODY[])
> ----------------
> 
> Randomly fetch message body from 100 messages
> 
> Dovecot:
> 
> ./imaptest - logout=0 select=100 fetch2=100 secs=10 seed=0 clients=1
> Logi Sele Fet2 
> 100% 100% 100% 
>            30% 
>    1    1 51958 
> ./imaptest - logout=0 select=100 fetch2=100 secs=10 seed=0 clients=5
> Logi Sele Fet2 
> 100% 100% 100% 
>            30% 
>    5    5 141397 
> 
> Zimbra:
> 
> ./imaptest - logout=0 select=100 fetch2=100 secs=10 seed=0 clients=1
> Logi Sele Fet2 
> 100% 100% 100% 
>            30% 
>    1    1  410 
> ./imaptest - logout=0 select=100 fetch2=100 secs=10 seed=0 clients=5
> Logi Sele Fet2 
> 100% 100% 100% 
>            30% 
>    5    5 2291 
> 
> STORE ? FLAGS.SILENT (random flags and keywords)
> ------------------------------------------------
> 
> Dovecot:
> 
> ./imaptest - logout=0 select=100 store=100 secs=10 seed=0 clients=1
> Logi Sele Stor 
> 100% 100% 100% 
>    1    1 15236 
> ./imaptest - logout=0 select=100 store=100 secs=10 seed=0 clients=5
> Logi Sele Stor 
> 100% 100% 100% 
>    5    5 5533 
> 
> Zimbra:
> 
> ./imaptest - logout=0 select=100 store=100 secs=10 seed=0 clients=1
> Logi Sele Stor 
> 100% 100% 100% 
>    1    1  308 
> ./imaptest - logout=0 select=100 store=100 secs=10 seed=0 clients=5
> Logi Sele Stor 
> 100% 100% 100% 
>    5    5  319 
> 
> Zimbra gave lots of errors:
> Error: STORE failed: STORE failed
> 
> APPEND
> ------
> 
> Appends to empty mailbox:
> 
> Dovecot:
> 1 client, 5 seconds: 3959 mails
> 1 client, 5 seconds, fsync_disable=yes: 21889 mails
> 
> Zimbra:
> 1 clients, 5 seconds: 222 mails
> 5 clients, 5 seconds: 322 mails
> 
> Zimbra is apparently building full text search indexes while appending,
> so this test doesn't mean much until I can test Dovecot's performance
> with Squat indexing.


-- 
  Levente                               "Si vis pacem para bellum!"