[Dovecot] dbmail benchmarking
I thought I'd try benchmarking with dbmail (v2.2.4) to see how much
slower a SQL backend could actually be. Skip to bottom for the
conclusions.
Originally I ran the tests with the databases being in XFS
filesystem. MySQL's performance was horrible. It went 3-7x faster
with ext3.
MySQL 5.0.30 backend (innodb):
./imaptest clients=1 - append=100 seed=1 secs=30 msgs=1000000 logout=0 Logi Sele Appe 100% 100% 100% 5% 1 291 303
So that's 10 messages/sec saved. Now how about with 5 concurrent
clients?
Logi Sele Appe 100% 100% 100% 5% 5 1259 1332
Pretty well. Then something more generic:
./imaptest clients=1 seed=1 secs=30 msgs=1000000 logout=0 Logi List Stat Sele Fetc Fet2 Stor Dele Expu Appe 100% 50% 50% 100% 100% 100% 50% 100% 100% 100% 30% 5% 1 37 36 75 73 110 34 24 73 78
Without Fetc (header/metadata fetching): Logi List Stat Sele Fet2 Stor Dele Expu Appe 100% 50% 50% 100% 100% 50% 100% 100% 100% 30% 5% 1 94 94 199 283 102 85 198 210
PostgreSQL 8.1.5 backend:
./imaptest clients=1 - append=100 seed=1 secs=30 msgs=1000000 logout=0 Logi Sele Appe 100% 100% 100% 5% 1 267 277
./imaptest clients=5 - append=100 seed=1 secs=30 msgs=1000000 logout=0 Logi Sele Appe 100% 100% 100% 5% 5 1094 1144
./imaptest clients=1 seed=1 secs=30 msgs=1000000 logout=0 Logi List Stat Sele Fetc Fet2 Stor Dele Expu Appe 100% 50% 50% 100% 100% 100% 50% 100% 100% 100% 30% 5% 9 29 40 74 72 99 22 12 64 71
./imaptest clients=1 seed=1 secs=30 msgs=1000000 logout=0 fetch=0 Logi List Stat Sele Fet2 Stor Dele Expu Appe 100% 50% 50% 100% 100% 50% 100% 100% 100% 30% 5% 35 105 95 200 277 54 70 165 175
The last two tests gave "Unexpected tagged reply:" errors that I
didn't get with MySQL. So apparently there's some PostgreSQL-specific
bug in dbmail.
The same values with Dovecot 1.0 + maildir:
./imaptest clients=1 - append=100 seed=1 secs=30 msgs=1000000 logout=0 Logi Sele Appe 100% 100% 100% 5% 1 346 364
./imaptest clients=5 - append=100 seed=1 secs=30 msgs=1000000 logout=0 Logi Sele Appe 100% 100% 100% 5% 5 1408 1470
./imaptest clients=1 seed=1 secs=30 msgs=1000000 logout=0 Logi List Stat Sele Fetc Fet2 Stor Dele Expu Appe 100% 50% 50% 100% 100% 100% 50% 100% 100% 100% 30% 5% 1 130 133 259 258 382 123 127 257 271
./imaptest clients=1 seed=1 secs=30 msgs=1000000 logout=0 fetch=0 Logi List Stat Sele Fet2 Stor Dele Expu Appe 100% 50% 50% 100% 100% 50% 100% 100% 100% 30% 5% 1 155 163 339 478 169 175 338 354
So, what are the conclusions?
- In raw append speed dbmail is almost as fast as maildir.
- In raw read/write speed maildir is about 1,6 times faster
- When adding metadata fetches Dovecot is 4 times faster than
dbmail. This is most likely because dbmail doesn't have a cache
equivalent to dovecot.index.cache so it has to do the fetches the
slow way.
I would have liked to also run the generic tests with more than 1
client, but then I start hitting dbmail bugs and the test stops
(reported already to their bugtracker).
Could someone please update the dbmail wikipedia page with this info ;)
http://en.wikipedia.org/wiki/DBMail_IMAP_and_POP3_server
I suppose what would be nicer would be to extend the dovecot entry and reference that from the dbmail entry ;)
On Sat, May 12, 2007 at 03:16:05AM +0300, Timo Sirainen wrote:
I thought I'd try benchmarking with dbmail (v2.2.4) to see how much
slower a SQL backend could actually be. Skip to bottom for the
conclusions.Originally I ran the tests with the databases being in XFS
filesystem. MySQL's performance was horrible. It went 3-7x faster
with ext3.MySQL 5.0.30 backend (innodb):
./imaptest clients=1 - append=100 seed=1 secs=30 msgs=1000000 logout=0 Logi Sele Appe 100% 100% 100% 5% 1 291 303
So that's 10 messages/sec saved. Now how about with 5 concurrent
clients?Logi Sele Appe 100% 100% 100% 5% 5 1259 1332
Pretty well. Then something more generic:
./imaptest clients=1 seed=1 secs=30 msgs=1000000 logout=0 Logi List Stat Sele Fetc Fet2 Stor Dele Expu Appe 100% 50% 50% 100% 100% 100% 50% 100% 100% 100% 30% 5% 1 37 36 75 73 110 34 24 73 78
Without Fetc (header/metadata fetching): Logi List Stat Sele Fet2 Stor Dele Expu Appe 100% 50% 50% 100% 100% 50% 100% 100% 100% 30% 5% 1 94 94 199 283 102 85 198 210
PostgreSQL 8.1.5 backend:
./imaptest clients=1 - append=100 seed=1 secs=30 msgs=1000000 logout=0 Logi Sele Appe 100% 100% 100% 5% 1 267 277
./imaptest clients=5 - append=100 seed=1 secs=30 msgs=1000000 logout=0 Logi Sele Appe 100% 100% 100% 5% 5 1094 1144
./imaptest clients=1 seed=1 secs=30 msgs=1000000 logout=0 Logi List Stat Sele Fetc Fet2 Stor Dele Expu Appe 100% 50% 50% 100% 100% 100% 50% 100% 100% 100% 30% 5% 9 29 40 74 72 99 22 12 64 71
./imaptest clients=1 seed=1 secs=30 msgs=1000000 logout=0 fetch=0 Logi List Stat Sele Fet2 Stor Dele Expu Appe 100% 50% 50% 100% 100% 50% 100% 100% 100% 30% 5% 35 105 95 200 277 54 70 165 175
The last two tests gave "Unexpected tagged reply:" errors that I
didn't get with MySQL. So apparently there's some PostgreSQL-specific
bug in dbmail.The same values with Dovecot 1.0 + maildir:
./imaptest clients=1 - append=100 seed=1 secs=30 msgs=1000000 logout=0 Logi Sele Appe 100% 100% 100% 5% 1 346 364
./imaptest clients=5 - append=100 seed=1 secs=30 msgs=1000000 logout=0 Logi Sele Appe 100% 100% 100% 5% 5 1408 1470
./imaptest clients=1 seed=1 secs=30 msgs=1000000 logout=0 Logi List Stat Sele Fetc Fet2 Stor Dele Expu Appe 100% 50% 50% 100% 100% 100% 50% 100% 100% 100% 30% 5% 1 130 133 259 258 382 123 127 257 271
./imaptest clients=1 seed=1 secs=30 msgs=1000000 logout=0 fetch=0 Logi List Stat Sele Fet2 Stor Dele Expu Appe 100% 50% 50% 100% 100% 50% 100% 100% 100% 30% 5% 1 155 163 339 478 169 175 338 354
So, what are the conclusions?
- In raw append speed dbmail is almost as fast as maildir.
- In raw read/write speed maildir is about 1,6 times faster
- When adding metadata fetches Dovecot is 4 times faster than
dbmail. This is most likely because dbmail doesn't have a cache
equivalent to dovecot.index.cache so it has to do the fetches the
slow way.I would have liked to also run the generic tests with more than 1
client, but then I start hitting dbmail bugs and the test stops
(reported already to their bugtracker).
--
Troy Benjegerdes 'da hozer' hozer@hozed.org
Somone asked me why I work on this free (http://www.fsf.org/philosophy/) software stuff and not get a real job. Charles Shultz had the best answer:
"Why do musicians compose symphonies and poets write poems? They do it because life wouldn't have any meaning for them if they didn't. That's why I draw cartoons. It's my life." -- Charles Shultz
Timo Sirainen wrote:
I thought I'd try benchmarking with dbmail (v2.2.4) to see how much slower a SQL backend could actually be. Skip to bottom for the conclusions.
Originally I ran the tests with the databases being in XFS filesystem. MySQL's performance was horrible. It went 3-7x faster with ext3.
Timo, what was the hardware you used for this test?
- In raw append speed dbmail is almost as fast as maildir.
- In raw read/write speed maildir is about 1,6 times faster
- When adding metadata fetches Dovecot is 4 times faster than dbmail. This is most likely because dbmail doesn't have a cache equivalent to dovecot.index.cache so it has to do the fetches the slow way.
I would have liked to also run the generic tests with more than 1 client, but then I start hitting dbmail bugs and the test stops (reported already to their bugtracker).
On Mon, 2007-05-14 at 14:03 -0400, Justin McAleer wrote:
Timo Sirainen wrote:
I thought I'd try benchmarking with dbmail (v2.2.4) to see how much slower a SQL backend could actually be. Skip to bottom for the conclusions.
Originally I ran the tests with the databases being in XFS filesystem. MySQL's performance was horrible. It went 3-7x faster with ext3.
Timo, what was the hardware you used for this test?
Intel Core 2 6600, 2GB 800MHz memory, Asus P5B, 300GB PATA Seagate disk with XFS filesystem, ext3 created as a 1GB file inside XFS (once I get another disk I'll move to ext3 completely).
On Sat, 2007-05-12 at 03:16 +0300, Timo Sirainen wrote:
- In raw append speed dbmail is almost as fast as maildir.
- In raw read/write speed maildir is about 1,6 times faster
- When adding metadata fetches Dovecot is 4 times faster than
dbmail. This is most likely because dbmail doesn't have a cache
equivalent to dovecot.index.cache so it has to do the fetches the
slow way.
I think these are wrong actually. In those benchmarks the server was mainly just waiting for new APPEND commands, so the results were somewhat similar. Now lets try with 100 clients to see how it goes:
dbmail SVN 2_2_branch + MySQL:
./imaptest - append=100 seed=1 secs=30 msgs=1000000 logout=0 select=0 clients=100 Logi Appe 100% 100% 5% 100 1642
Hmm. Maybe that was too much for it. Lets try with 30:
./imaptest - append=100 seed=1 secs=30 msgs=1000000 logout=0 select=0 clients=30 Logi Appe 100% 100% 5% 30 2620
So the performance got better with less simultaneous clients.
Dovecot CVS HEAD + maildir + fsync_disable=no: ./imaptest - append=100 seed=1 secs=30 msgs=1000000 logout=0 select=0 clients=100 Logi Appe 100% 100% 5% 100 4720
The speed dropped from 200 msgs/sec to 100 msgs/sec. The larger the maildir, the slower it becomes. Lets try with 30 clients with Dovecot too:
./imaptest - append=100 seed=1 secs=30 msgs=1000000 logout=0 select=0 clients=30 Logi Appe 100% 100% 5% 30 3960
msgs/sec drops, as expected. Now, lets see how much fsync is slowing it down:
Dovecot CVS HEAD + maildir + fsync_disable=yes: ./imaptest - append=100 seed=1 secs=30 msgs=1000000 logout=0 select=0 clients=100 Logi Appe 100% 100% 5% 100 5951
The speed drops from 587/sec -> 113/sec. Lets try Dovecot's high performance Cyrus-like cydir format then:
Dovecot CVS HEAD + cydir + fsync_disable=no: ./imaptest - append=100 seed=1 secs=30 msgs=1000000 logout=0 select=0 clients=100 Logi Appe 100% 100% 5% 100 35217
Well, isn't that something. Almost 10 times faster than maildir. The speed only drops from 1240/sec to 1150/sec. fsync_disable=yes gives +1000 messages, so not that much.
Since the speed was quite steady 11-12/connection, lets try with some more connections. How about 200 clients:
Dovecot CVS HEAD + cydir + fsync_disable=no: ./imaptest - append=100 seed=1 secs=30 msgs=1000000 logout=0 select=0 clients=200 Logi Appe 100% 100% 5% 200 1398 200/200 0 2268 200/200 0 2145 200/200 0 1965 200/200 0 1448 200/200 0 1650 200/200 0 1216 200/200 0 1193 200/200 0 1112 200/200 ... 0 708 200/200 0 812 200/200 0 726 200/200 0 780 200/200 0 727 200/200
Totals: Logi Appe 100% 100% 5% 200 32844
This didn't go so well anymore. It started nicely with the expected 2200/sec, but dropped rapidly. Might be because of lock contention, or because of ext3, or even because of imaptest itself.
On Wed, 2007-05-16 at 21:21 +0300, Timo Sirainen wrote:
Dovecot CVS HEAD + cydir + fsync_disable=no: ./imaptest - append=100 seed=1 secs=30 msgs=1000000 logout=0 select=0 clients=100 Logi Appe 100% 100% 5% 100 35217
In the above tests the cache file wasn't updated at all, so it's a bit unrealistic test. Lets try again with ENVELOPE headers, BODYSTRUCTURE and INTERNALDATE added into the cache file.
Dovecot CVS HEAD + cydir + fsync_disable=no: ./imaptest - append=100 seed=1 secs=30 msgs=1000000 logout=0 select=0 clients=100 100 1023 100/100 0 1204 100/100 0 1109 100/100 0 1104 100/100 0 976 100/100 0 824 100/100 0 781 100/100 0 762 100/100 .. 0 306 100/100 0 369 100/100
Totals: Logi Appe 100% 100% 5% 100 18032
So, dovecot.index.cache file could work more nicely with larger mailboxes. Hmm. I can't seem to figure out easily why it's slowing down that much though. I guess I'll have to look at it more.
Oh, and cydir is pretty much the same as what Cyrus is using internally. The only differences are how our index/cache files are implemented. Here's how Cyrus 2.2.13-10+b1 (Debian) looks like:
./imaptest - append=100 seed=1 secs=30 msgs=1000000 logout=0 select=0 clients=100 Logi Appe 100% 100% 5% 100 8546
It was pretty constant at 280/sec. So Dovecot is still about twice as fast at appending, at least up to 18k mail mailboxes. Cyrus writes a bit more into its cache file, but it shouldn't give this big of a difference.
participants (3)
-
Justin McAleer
-
Timo Sirainen
-
Troy Benjegerdes