[Dovecot] Zimbra benchmarking

[sergey ivanov]

Hi,
for me zimbra swapped a lot on 1G of RAM even while serving mailboxes
for 3 users. But on 2G it runs such that I can't feel it's slower than
previously worked there dovecot.
-- 
    Sergey.

Timo Sirainen wrote:
> Now that I have a working kvm setup, I thought I'd finally try how
> Zimbra works. This is mainly some microbenchmarking, so it may not have
> much to do with actual performance in real life.
>
> Setup:
>
>  - 1GB memory given to kvm (from host's 2GB)
>  - Intel Core 2 6600 (kvm uses only one CPU)
>  - CentOS 5
>  - 15GB qcow2 image on XFS filesystem
>  - Zimbra 5.0 RC2 RHEL5 x86_64
>  - Dovecot latest hg, dbox format
>
> Both Zimbra and Dovecot were tested running on the same kvm image. All
> data could be cached in memory, so this didn't really test disk I/O.
>
> Dovecot was run with fsync_disable=no. Running with =yes would have made
> its performance even better.
>
> Zimbra features/bugs
> --------------------
>
> Zimbra's SEARCH command doesn't support substring searching as IMAP
> requires:
>
> 1 search text performance
> * SEARCH 136 183 227 231 232 233 245
> 1 OK SEARCH completed
> 2 search text erformance
> * SEARCH
> 2 OK SEARCH completed
> 3 search text performanc
> * SEARCH
> 3 OK SEARCH completed
>
> It's possible to search "performanc" in a non-standard way though:
>
> 4 search text "performanc*"
> * SEARCH 136 183 227 231 232 233 245
> 4 OK SEARCH completed
>
> This means that the upcoming Dovecot v1.1 is probably the only IMAP
> server that supports IMAP-compatible full text search indexes with
> incremental updates. (Cyrus Squat requires rebuilding the whole index
> from scratch just for adding one mail, which makes it kind of
> impractical.)
>
> Zimbra doesn't support SORT or THREAD extensions.
>
> I had trouble benchmarking more than 5 simultaneous connections in one
> mailbox. I'm not sure if this is a configurable setting somewhere, or if
> there's just some bug. The error messages that my imaptest gave looked
> like Zimbra was buggy, but I didn't look at them too closely.
>
> STORE command also seemed to be buggy, giving lots of "STORE failed"
> errors, even with just one connection.
>
> Zimbra uses 500MB of memory just to start up, so it's not exactly for
> hosting small installations.
>
> Login+Logout
> ------------
>
> Dovecot:
>
> ./imaptest - select=0 secs=10 seed=0 clients=1
> Logi Logo 
> 100% 100% 
> 3439 6878 
> ./imaptest - select=0 secs=10 seed=0 clients=10
> Logi Logo 
> 100% 100% 
> 4415 8830 
>
> Zimbra:
>
> ./imaptest - select=0 secs=10 seed=0 clients=1
> Logi Logo 
> 100% 100% 
> 18032 36064 
> ./imaptest - select=0 secs=10 seed=0 clients=10
> Logi Logo 
> 100% 100% 
> 25519 51056 
>
> Looks like Dovecot's imap process creation hurts it a lot compared to
> Zimbra's thread creation. Luckily IMAP connections are usually
> long-living, so this shouldn't matter much, except for webmails for
> which you can use imapproxy in the middle.
>
> LIST "" *
> ---------
>
> Dovecot:
>
> ./imaptest - select=0 logout=0 list=100 secs=10 seed=0 clients=1
> Logi List 
> 100% 100% 
>    1 181411 
> ./imaptest - select=0 logout=0 list=100 secs=10 seed=0 clients=10
> Logi List 
> 100% 100% 
>   10 133451 
>
> Zimbra:
>
> ./imaptest - select=0 logout=0 list=100 secs=10 seed=0 clients=1
> Logi List 
> 100% 100% 
>    1 1021 
> ./imaptest - select=0 logout=0 list=100 secs=10 seed=0 clients=10
> Logi List 
> 100% 100% 
>   10  972 
>
> Yes, Dovecot's LIST is over 100 times faster.
>
> STATUS INBOX (MESSAGES UNSEEN RECENT)
> -------------------------------------
>
> 100 messages in INBOX 
>
> dovecot:
>
> ./imaptest - logout=0 select=0 status=100 secs=10 seed=0 clients=1
> Logi Stat 
> 100% 100% 
>    1 191003 
> ./imaptest - logout=0 select=0 status=100 secs=10 seed=0 clients=10
> Logi Stat 
> 100% 100% 
>   10 171171 
>
> Zimbra:
>
> ./imaptest - logout=0 select=0 status=100 secs=10 seed=0 clients=1
> Logi Stat 
> 100% 100% 
>    1 3697 
> ./imaptest - logout=0 select=0 status=100 secs=10 seed=0 clients=10
> Logi Stat 
> 100% 100% 
>   10 7009 
>
> FETCH 1:* (UID FLAGS ENVELOPE INTERNALDATE BODYSTRUCTURE)
> ---------------------------------------------------------
>
> 100 messages in INBOX.
>
> dovecot:
> ./imaptest - logout=0 select=100 fetch=100 secs=10 seed=0 clients=1
> Logi Sele Fetc 
> 100% 100% 100% 
>    1    1 2769 
> ./imaptest - logout=0 select=100 fetch=100 secs=10 seed=0 clients=5
> Logi Sele Fetc 
> 100% 100% 100% 
>    5    5 2902 
>
> Zimbra:
>
> ./imaptest - logout=0 select=100 fetch=100 secs=10 seed=0 clients=1
> Logi Sele Fetc 
> 100% 100% 100% 
>    1    1  203 
> ./imaptest - logout=0 select=100 fetch=100 secs=10 seed=0 clients=5
> Logi Sele Fetc 
> 100% 100% 100% 
>    5    5  258 
>
> FETCH ? (BODY[])
> ----------------
>
> Randomly fetch message body from 100 messages
>
> Dovecot:
>
> ./imaptest - logout=0 select=100 fetch2=100 secs=10 seed=0 clients=1
> Logi Sele Fet2 
> 100% 100% 100% 
>            30% 
>    1    1 51958 
> ./imaptest - logout=0 select=100 fetch2=100 secs=10 seed=0 clients=5
> Logi Sele Fet2 
> 100% 100% 100% 
>            30% 
>    5    5 141397 
>
> Zimbra:
>
> ./imaptest - logout=0 select=100 fetch2=100 secs=10 seed=0 clients=1
> Logi Sele Fet2 
> 100% 100% 100% 
>            30% 
>    1    1  410 
> ./imaptest - logout=0 select=100 fetch2=100 secs=10 seed=0 clients=5
> Logi Sele Fet2 
> 100% 100% 100% 
>            30% 
>    5    5 2291 
>
> STORE ? FLAGS.SILENT (random flags and keywords)
> ------------------------------------------------
>
> Dovecot:
>
> ./imaptest - logout=0 select=100 store=100 secs=10 seed=0 clients=1
> Logi Sele Stor 
> 100% 100% 100% 
>    1    1 15236 
> ./imaptest - logout=0 select=100 store=100 secs=10 seed=0 clients=5
> Logi Sele Stor 
> 100% 100% 100% 
>    5    5 5533 
>
> Zimbra:
>
> ./imaptest - logout=0 select=100 store=100 secs=10 seed=0 clients=1
> Logi Sele Stor 
> 100% 100% 100% 
>    1    1  308 
> ./imaptest - logout=0 select=100 store=100 secs=10 seed=0 clients=5
> Logi Sele Stor 
> 100% 100% 100% 
>    5    5  319 
>
> Zimbra gave lots of errors:
> Error: STORE failed: STORE failed
>
> APPEND
> ------
>
> Appends to empty mailbox:
>
> Dovecot:
> 1 client, 5 seconds: 3959 mails
> 1 client, 5 seconds, fsync_disable=yes: 21889 mails
>
> Zimbra:
> 1 clients, 5 seconds: 222 mails
> 5 clients, 5 seconds: 322 mails
>
> Zimbra is apparently building full text search indexes while appending,
> so this test doesn't mean much until I can test Dovecot's performance
> with Squat indexing.
>