Now that I have a working kvm setup, I thought I'd finally try how Zimbra works. This is mainly some microbenchmarking, so it may not have much to do with actual performance in real life.
Setup:
- 1GB memory given to kvm (from host's 2GB)
- Intel Core 2 6600 (kvm uses only one CPU)
- CentOS 5
- 15GB qcow2 image on XFS filesystem
- Zimbra 5.0 RC2 RHEL5 x86_64
- Dovecot latest hg, dbox format
Both Zimbra and Dovecot were tested running on the same kvm image. All data could be cached in memory, so this didn't really test disk I/O.
Dovecot was run with fsync_disable=no. Running with =yes would have made its performance even better.
Zimbra features/bugs
Zimbra's SEARCH command doesn't support substring searching as IMAP requires:
1 search text performance
- SEARCH 136 183 227 231 232 233 245 1 OK SEARCH completed 2 search text erformance
- SEARCH 2 OK SEARCH completed 3 search text performanc
- SEARCH 3 OK SEARCH completed
It's possible to search "performanc" in a non-standard way though:
4 search text "performanc*"
- SEARCH 136 183 227 231 232 233 245 4 OK SEARCH completed
This means that the upcoming Dovecot v1.1 is probably the only IMAP server that supports IMAP-compatible full text search indexes with incremental updates. (Cyrus Squat requires rebuilding the whole index from scratch just for adding one mail, which makes it kind of impractical.)
Zimbra doesn't support SORT or THREAD extensions.
I had trouble benchmarking more than 5 simultaneous connections in one mailbox. I'm not sure if this is a configurable setting somewhere, or if there's just some bug. The error messages that my imaptest gave looked like Zimbra was buggy, but I didn't look at them too closely.
STORE command also seemed to be buggy, giving lots of "STORE failed" errors, even with just one connection.
Zimbra uses 500MB of memory just to start up, so it's not exactly for hosting small installations.
Login+Logout
Dovecot:
./imaptest - select=0 secs=10 seed=0 clients=1 Logi Logo 100% 100% 3439 6878 ./imaptest - select=0 secs=10 seed=0 clients=10 Logi Logo 100% 100% 4415 8830
Zimbra:
./imaptest - select=0 secs=10 seed=0 clients=1 Logi Logo 100% 100% 18032 36064 ./imaptest - select=0 secs=10 seed=0 clients=10 Logi Logo 100% 100% 25519 51056
Looks like Dovecot's imap process creation hurts it a lot compared to Zimbra's thread creation. Luckily IMAP connections are usually long-living, so this shouldn't matter much, except for webmails for which you can use imapproxy in the middle.
LIST "" *
Dovecot:
./imaptest - select=0 logout=0 list=100 secs=10 seed=0 clients=1 Logi List 100% 100% 1 181411 ./imaptest - select=0 logout=0 list=100 secs=10 seed=0 clients=10 Logi List 100% 100% 10 133451
Zimbra:
./imaptest - select=0 logout=0 list=100 secs=10 seed=0 clients=1 Logi List 100% 100% 1 1021 ./imaptest - select=0 logout=0 list=100 secs=10 seed=0 clients=10 Logi List 100% 100% 10 972
Yes, Dovecot's LIST is over 100 times faster.
STATUS INBOX (MESSAGES UNSEEN RECENT)
100 messages in INBOX
dovecot:
./imaptest - logout=0 select=0 status=100 secs=10 seed=0 clients=1 Logi Stat 100% 100% 1 191003 ./imaptest - logout=0 select=0 status=100 secs=10 seed=0 clients=10 Logi Stat 100% 100% 10 171171
Zimbra:
./imaptest - logout=0 select=0 status=100 secs=10 seed=0 clients=1 Logi Stat 100% 100% 1 3697 ./imaptest - logout=0 select=0 status=100 secs=10 seed=0 clients=10 Logi Stat 100% 100% 10 7009
FETCH 1:* (UID FLAGS ENVELOPE INTERNALDATE BODYSTRUCTURE)
100 messages in INBOX.
dovecot: ./imaptest - logout=0 select=100 fetch=100 secs=10 seed=0 clients=1 Logi Sele Fetc 100% 100% 100% 1 1 2769 ./imaptest - logout=0 select=100 fetch=100 secs=10 seed=0 clients=5 Logi Sele Fetc 100% 100% 100% 5 5 2902
Zimbra:
./imaptest - logout=0 select=100 fetch=100 secs=10 seed=0 clients=1 Logi Sele Fetc 100% 100% 100% 1 1 203 ./imaptest - logout=0 select=100 fetch=100 secs=10 seed=0 clients=5 Logi Sele Fetc 100% 100% 100% 5 5 258
FETCH ? (BODY[])
Randomly fetch message body from 100 messages
Dovecot:
./imaptest - logout=0 select=100 fetch2=100 secs=10 seed=0 clients=1 Logi Sele Fet2 100% 100% 100% 30% 1 1 51958 ./imaptest - logout=0 select=100 fetch2=100 secs=10 seed=0 clients=5 Logi Sele Fet2 100% 100% 100% 30% 5 5 141397
Zimbra:
./imaptest - logout=0 select=100 fetch2=100 secs=10 seed=0 clients=1 Logi Sele Fet2 100% 100% 100% 30% 1 1 410 ./imaptest - logout=0 select=100 fetch2=100 secs=10 seed=0 clients=5 Logi Sele Fet2 100% 100% 100% 30% 5 5 2291
STORE ? FLAGS.SILENT (random flags and keywords)
Dovecot:
./imaptest - logout=0 select=100 store=100 secs=10 seed=0 clients=1 Logi Sele Stor 100% 100% 100% 1 1 15236 ./imaptest - logout=0 select=100 store=100 secs=10 seed=0 clients=5 Logi Sele Stor 100% 100% 100% 5 5 5533
Zimbra:
./imaptest - logout=0 select=100 store=100 secs=10 seed=0 clients=1 Logi Sele Stor 100% 100% 100% 1 1 308 ./imaptest - logout=0 select=100 store=100 secs=10 seed=0 clients=5 Logi Sele Stor 100% 100% 100% 5 5 319
Zimbra gave lots of errors: Error: STORE failed: STORE failed
APPEND
Appends to empty mailbox:
Dovecot: 1 client, 5 seconds: 3959 mails 1 client, 5 seconds, fsync_disable=yes: 21889 mails
Zimbra: 1 clients, 5 seconds: 222 mails 5 clients, 5 seconds: 322 mails
Zimbra is apparently building full text search indexes while appending, so this test doesn't mean much until I can test Dovecot's performance with Squat indexing.
hi, nice test, but what would be also useful to compare cpu and memory usage! with zimbra it's more important then anything else. i'd like to see a comparison what kind of hardware (cpu and memory) required for: 10, 100, 500, 1000, 2000, 5000 mailbox server. this's where dovecot a big winner even for 10 mailbox. but to be fair zimbra is much more than dovecot it's integrate many things (ldap user and group management, samba domain controller, virus scanner, outlook connector (!) not just for email but also for calendar and address book, etc...) and to put together such a system with dovecot it's more than hard and may be impossible.
Timo Sirainen wrote:
Now that I have a working kvm setup, I thought I'd finally try how Zimbra works. This is mainly some microbenchmarking, so it may not have much to do with actual performance in real life.
Setup:
- 1GB memory given to kvm (from host's 2GB)
- Intel Core 2 6600 (kvm uses only one CPU)
- CentOS 5
- 15GB qcow2 image on XFS filesystem
- Zimbra 5.0 RC2 RHEL5 x86_64
- Dovecot latest hg, dbox format
Both Zimbra and Dovecot were tested running on the same kvm image. All data could be cached in memory, so this didn't really test disk I/O.
Dovecot was run with fsync_disable=no. Running with =yes would have made its performance even better.
Zimbra features/bugs
Zimbra's SEARCH command doesn't support substring searching as IMAP requires:
1 search text performance
- SEARCH 136 183 227 231 232 233 245 1 OK SEARCH completed 2 search text erformance
- SEARCH 2 OK SEARCH completed 3 search text performanc
- SEARCH 3 OK SEARCH completed
It's possible to search "performanc" in a non-standard way though:
4 search text "performanc*"
- SEARCH 136 183 227 231 232 233 245 4 OK SEARCH completed
This means that the upcoming Dovecot v1.1 is probably the only IMAP server that supports IMAP-compatible full text search indexes with incremental updates. (Cyrus Squat requires rebuilding the whole index from scratch just for adding one mail, which makes it kind of impractical.)
Zimbra doesn't support SORT or THREAD extensions.
I had trouble benchmarking more than 5 simultaneous connections in one mailbox. I'm not sure if this is a configurable setting somewhere, or if there's just some bug. The error messages that my imaptest gave looked like Zimbra was buggy, but I didn't look at them too closely.
STORE command also seemed to be buggy, giving lots of "STORE failed" errors, even with just one connection.
Zimbra uses 500MB of memory just to start up, so it's not exactly for hosting small installations.
Login+Logout
Dovecot:
./imaptest - select=0 secs=10 seed=0 clients=1 Logi Logo 100% 100% 3439 6878 ./imaptest - select=0 secs=10 seed=0 clients=10 Logi Logo 100% 100% 4415 8830
Zimbra:
./imaptest - select=0 secs=10 seed=0 clients=1 Logi Logo 100% 100% 18032 36064 ./imaptest - select=0 secs=10 seed=0 clients=10 Logi Logo 100% 100% 25519 51056
Looks like Dovecot's imap process creation hurts it a lot compared to Zimbra's thread creation. Luckily IMAP connections are usually long-living, so this shouldn't matter much, except for webmails for which you can use imapproxy in the middle.
LIST "" *
Dovecot:
./imaptest - select=0 logout=0 list=100 secs=10 seed=0 clients=1 Logi List 100% 100% 1 181411 ./imaptest - select=0 logout=0 list=100 secs=10 seed=0 clients=10 Logi List 100% 100% 10 133451
Zimbra:
./imaptest - select=0 logout=0 list=100 secs=10 seed=0 clients=1 Logi List 100% 100% 1 1021 ./imaptest - select=0 logout=0 list=100 secs=10 seed=0 clients=10 Logi List 100% 100% 10 972
Yes, Dovecot's LIST is over 100 times faster.
STATUS INBOX (MESSAGES UNSEEN RECENT)
100 messages in INBOX
dovecot:
./imaptest - logout=0 select=0 status=100 secs=10 seed=0 clients=1 Logi Stat 100% 100% 1 191003 ./imaptest - logout=0 select=0 status=100 secs=10 seed=0 clients=10 Logi Stat 100% 100% 10 171171
Zimbra:
./imaptest - logout=0 select=0 status=100 secs=10 seed=0 clients=1 Logi Stat 100% 100% 1 3697 ./imaptest - logout=0 select=0 status=100 secs=10 seed=0 clients=10 Logi Stat 100% 100% 10 7009
FETCH 1:* (UID FLAGS ENVELOPE INTERNALDATE BODYSTRUCTURE)
100 messages in INBOX.
dovecot: ./imaptest - logout=0 select=100 fetch=100 secs=10 seed=0 clients=1 Logi Sele Fetc 100% 100% 100% 1 1 2769 ./imaptest - logout=0 select=100 fetch=100 secs=10 seed=0 clients=5 Logi Sele Fetc 100% 100% 100% 5 5 2902
Zimbra:
./imaptest - logout=0 select=100 fetch=100 secs=10 seed=0 clients=1 Logi Sele Fetc 100% 100% 100% 1 1 203 ./imaptest - logout=0 select=100 fetch=100 secs=10 seed=0 clients=5 Logi Sele Fetc 100% 100% 100% 5 5 258
FETCH ? (BODY[])
Randomly fetch message body from 100 messages
Dovecot:
./imaptest - logout=0 select=100 fetch2=100 secs=10 seed=0 clients=1 Logi Sele Fet2 100% 100% 100% 30% 1 1 51958 ./imaptest - logout=0 select=100 fetch2=100 secs=10 seed=0 clients=5 Logi Sele Fet2 100% 100% 100% 30% 5 5 141397
Zimbra:
./imaptest - logout=0 select=100 fetch2=100 secs=10 seed=0 clients=1 Logi Sele Fet2 100% 100% 100% 30% 1 1 410 ./imaptest - logout=0 select=100 fetch2=100 secs=10 seed=0 clients=5 Logi Sele Fet2 100% 100% 100% 30% 5 5 2291
STORE ? FLAGS.SILENT (random flags and keywords)
Dovecot:
./imaptest - logout=0 select=100 store=100 secs=10 seed=0 clients=1 Logi Sele Stor 100% 100% 100% 1 1 15236 ./imaptest - logout=0 select=100 store=100 secs=10 seed=0 clients=5 Logi Sele Stor 100% 100% 100% 5 5 5533
Zimbra:
./imaptest - logout=0 select=100 store=100 secs=10 seed=0 clients=1 Logi Sele Stor 100% 100% 100% 1 1 308 ./imaptest - logout=0 select=100 store=100 secs=10 seed=0 clients=5 Logi Sele Stor 100% 100% 100% 5 5 319
Zimbra gave lots of errors: Error: STORE failed: STORE failed
APPEND
Appends to empty mailbox:
Dovecot: 1 client, 5 seconds: 3959 mails 1 client, 5 seconds, fsync_disable=yes: 21889 mails
Zimbra: 1 clients, 5 seconds: 222 mails 5 clients, 5 seconds: 322 mails
Zimbra is apparently building full text search indexes while appending, so this test doesn't mean much until I can test Dovecot's performance with Squat indexing.
-- Levente "Si vis pacem para bellum!"
On Nov 30, 2007 11:38 AM, Timo Sirainen tss@iki.fi wrote:
Zimbra is apparently building full text search indexes while appending, so this test doesn't mean much until I can test Dovecot's performance with Squat indexing.
Hello, in fact, I'm not that much convinced by full-text search index server side. We consider response time, server side full text search will include client-server round-trip. So that, for example, on etpanX, I do some local indexing on the imap folders, so that when the user does a search, it's given in a fraction of second even if the server is slow. I think Mail.app on Mac OS X is doing the same.
-- DINH Viêt Hoà
On Fri, 2007-11-30 at 12:00 +0100, DINH Viêt Hoà wrote:
On Nov 30, 2007 11:38 AM, Timo Sirainen tss@iki.fi wrote:
Zimbra is apparently building full text search indexes while appending, so this test doesn't mean much until I can test Dovecot's performance with Squat indexing.
Hello, in fact, I'm not that much convinced by full-text search index server side. We consider response time, server side full text search will include client-server round-trip. So that, for example, on etpanX, I do some local indexing on the imap folders, so that when the user does a search, it's given in a fraction of second even if the server is slow. I think Mail.app on Mac OS X is doing the same.
Yes, and so do many other clients. But FTS indexes are useful for webmails, mobile clients and other clients that don't have a local cache.
Squat indexes are going to work so that the first time you do a TEXT or BODY search the indexes are built for the mailbox. After that they most likely are updated when saving new mails (although maybe only with deliver, not with APPEND/COPY?). If user hasn't done any TEXT/BODY searches for a month or so, the indexes finally get deleted. Or that's my plan currently, those rules are easy to change.
On Fri, Nov 30, 2007 at 01:16:25PM +0200, Timo Sirainen wrote:
On Fri, 2007-11-30 at 12:00 +0100, DINH Vi?t Ho? wrote:
On Nov 30, 2007 11:38 AM, Timo Sirainen tss@iki.fi wrote:
Zimbra is apparently building full text search indexes while appending, so this test doesn't mean much until I can test Dovecot's performance with Squat indexing.
Hello, in fact, I'm not that much convinced by full-text search index server side. We consider response time, server side full text search will include client-server round-trip. So that, for example, on etpanX, I do some local indexing on the imap folders, so that when the user does a search, it's given in a fraction of second even if the server is slow. I think Mail.app on Mac OS X is doing the same.
Yes, and so do many other clients. But FTS indexes are useful for webmails, mobile clients and other clients that don't have a local cache.
Squat indexes are going to work so that the first time you do a TEXT or BODY search the indexes are built for the mailbox. After that they most likely are updated when saving new mails (although maybe only with deliver, not with APPEND/COPY?). If user hasn't done any TEXT/BODY searches for a month or so, the indexes finally get deleted. Or that's my plan currently, those rules are easy to change.
I (and other users here) have some mail folder archives that go back several years and a number of those folders won't change, I what I was thinking of doing was do a full text search over ALL of the mail I can possibly access, so I will be all set for future fast searches, which might be of interest only once a year. I was thinking after we get dovecot into full production for at least half or a whole year, evaluating the space used by indexes of any type and make a site specific purge script if desired. It sounds like the indexes, if not automatically deleted, will be worth the more or less permanent disk space for us. Sounds like it would be useful as a configurable option.
On Nov 30, 2007 12:00 PM, DINH Viêt Hoà dinh.viet.hoa@free.fr wrote:
On Nov 30, 2007 11:38 AM, Timo Sirainen tss@iki.fi wrote:
Zimbra is apparently building full text search indexes while appending, so this test doesn't mean much until I can test Dovecot's performance with Squat indexing.
Hello, in fact, I'm not that much convinced by full-text search index server side. We consider response time, server side full text search will include client-server round-trip. So that, for example, on etpanX, I do some local indexing on the imap folders, so that when the user does a search, it's given in a fraction of second even if the server is slow. I think Mail.app on Mac OS X is doing the same.
By the way, how does Dovecot index the messsages ? Does it do the following ?
- when doing the search, it will collect messages that were not indexed
- will index those messages
- and do the search on the updated index.
-- DINH Viêt Hoà
Hi, for me zimbra swapped a lot on 1G of RAM even while serving mailboxes for 3 users. But on 2G it runs such that I can't feel it's slower than previously worked there dovecot.
Sergey.Timo Sirainen wrote:
Now that I have a working kvm setup, I thought I'd finally try how Zimbra works. This is mainly some microbenchmarking, so it may not have much to do with actual performance in real life.
Setup:
- 1GB memory given to kvm (from host's 2GB)
- Intel Core 2 6600 (kvm uses only one CPU)
- CentOS 5
- 15GB qcow2 image on XFS filesystem
- Zimbra 5.0 RC2 RHEL5 x86_64
- Dovecot latest hg, dbox format
Both Zimbra and Dovecot were tested running on the same kvm image. All data could be cached in memory, so this didn't really test disk I/O.
Dovecot was run with fsync_disable=no. Running with =yes would have made its performance even better.
Zimbra features/bugs
Zimbra's SEARCH command doesn't support substring searching as IMAP requires:
1 search text performance
- SEARCH 136 183 227 231 232 233 245 1 OK SEARCH completed 2 search text erformance
- SEARCH 2 OK SEARCH completed 3 search text performanc
- SEARCH 3 OK SEARCH completed
It's possible to search "performanc" in a non-standard way though:
4 search text "performanc*"
- SEARCH 136 183 227 231 232 233 245 4 OK SEARCH completed
This means that the upcoming Dovecot v1.1 is probably the only IMAP server that supports IMAP-compatible full text search indexes with incremental updates. (Cyrus Squat requires rebuilding the whole index from scratch just for adding one mail, which makes it kind of impractical.)
Zimbra doesn't support SORT or THREAD extensions.
I had trouble benchmarking more than 5 simultaneous connections in one mailbox. I'm not sure if this is a configurable setting somewhere, or if there's just some bug. The error messages that my imaptest gave looked like Zimbra was buggy, but I didn't look at them too closely.
STORE command also seemed to be buggy, giving lots of "STORE failed" errors, even with just one connection.
Zimbra uses 500MB of memory just to start up, so it's not exactly for hosting small installations.
Login+Logout
Dovecot:
./imaptest - select=0 secs=10 seed=0 clients=1 Logi Logo 100% 100% 3439 6878 ./imaptest - select=0 secs=10 seed=0 clients=10 Logi Logo 100% 100% 4415 8830
Zimbra:
./imaptest - select=0 secs=10 seed=0 clients=1 Logi Logo 100% 100% 18032 36064 ./imaptest - select=0 secs=10 seed=0 clients=10 Logi Logo 100% 100% 25519 51056
Looks like Dovecot's imap process creation hurts it a lot compared to Zimbra's thread creation. Luckily IMAP connections are usually long-living, so this shouldn't matter much, except for webmails for which you can use imapproxy in the middle.
LIST "" *
Dovecot:
./imaptest - select=0 logout=0 list=100 secs=10 seed=0 clients=1 Logi List 100% 100% 1 181411 ./imaptest - select=0 logout=0 list=100 secs=10 seed=0 clients=10 Logi List 100% 100% 10 133451
Zimbra:
./imaptest - select=0 logout=0 list=100 secs=10 seed=0 clients=1 Logi List 100% 100% 1 1021 ./imaptest - select=0 logout=0 list=100 secs=10 seed=0 clients=10 Logi List 100% 100% 10 972
Yes, Dovecot's LIST is over 100 times faster.
STATUS INBOX (MESSAGES UNSEEN RECENT)
100 messages in INBOX
dovecot:
./imaptest - logout=0 select=0 status=100 secs=10 seed=0 clients=1 Logi Stat 100% 100% 1 191003 ./imaptest - logout=0 select=0 status=100 secs=10 seed=0 clients=10 Logi Stat 100% 100% 10 171171
Zimbra:
./imaptest - logout=0 select=0 status=100 secs=10 seed=0 clients=1 Logi Stat 100% 100% 1 3697 ./imaptest - logout=0 select=0 status=100 secs=10 seed=0 clients=10 Logi Stat 100% 100% 10 7009
FETCH 1:* (UID FLAGS ENVELOPE INTERNALDATE BODYSTRUCTURE)
100 messages in INBOX.
dovecot: ./imaptest - logout=0 select=100 fetch=100 secs=10 seed=0 clients=1 Logi Sele Fetc 100% 100% 100% 1 1 2769 ./imaptest - logout=0 select=100 fetch=100 secs=10 seed=0 clients=5 Logi Sele Fetc 100% 100% 100% 5 5 2902
Zimbra:
./imaptest - logout=0 select=100 fetch=100 secs=10 seed=0 clients=1 Logi Sele Fetc 100% 100% 100% 1 1 203 ./imaptest - logout=0 select=100 fetch=100 secs=10 seed=0 clients=5 Logi Sele Fetc 100% 100% 100% 5 5 258
FETCH ? (BODY[])
Randomly fetch message body from 100 messages
Dovecot:
./imaptest - logout=0 select=100 fetch2=100 secs=10 seed=0 clients=1 Logi Sele Fet2 100% 100% 100% 30% 1 1 51958 ./imaptest - logout=0 select=100 fetch2=100 secs=10 seed=0 clients=5 Logi Sele Fet2 100% 100% 100% 30% 5 5 141397
Zimbra:
./imaptest - logout=0 select=100 fetch2=100 secs=10 seed=0 clients=1 Logi Sele Fet2 100% 100% 100% 30% 1 1 410 ./imaptest - logout=0 select=100 fetch2=100 secs=10 seed=0 clients=5 Logi Sele Fet2 100% 100% 100% 30% 5 5 2291
STORE ? FLAGS.SILENT (random flags and keywords)
Dovecot:
./imaptest - logout=0 select=100 store=100 secs=10 seed=0 clients=1 Logi Sele Stor 100% 100% 100% 1 1 15236 ./imaptest - logout=0 select=100 store=100 secs=10 seed=0 clients=5 Logi Sele Stor 100% 100% 100% 5 5 5533
Zimbra:
./imaptest - logout=0 select=100 store=100 secs=10 seed=0 clients=1 Logi Sele Stor 100% 100% 100% 1 1 308 ./imaptest - logout=0 select=100 store=100 secs=10 seed=0 clients=5 Logi Sele Stor 100% 100% 100% 5 5 319
Zimbra gave lots of errors: Error: STORE failed: STORE failed
APPEND
Appends to empty mailbox:
Dovecot: 1 client, 5 seconds: 3959 mails 1 client, 5 seconds, fsync_disable=yes: 21889 mails
Zimbra: 1 clients, 5 seconds: 222 mails 5 clients, 5 seconds: 322 mails
Zimbra is apparently building full text search indexes while appending, so this test doesn't mean much until I can test Dovecot's performance with Squat indexing.
participants (5)
-
Adam McDougall
-
DINH Viêt Hoà
-
Farkas Levente
-
sergey ivanov
-
Timo Sirainen