[Dovecot] uiddir mailbox format with benchmarks

[Timo Sirainen]

I cleaned up the lib-storage code a bit so that it's easier to write
support for new mailbox formats. It could still use a bit of cleaning
up.

Anyway, I thought I'd try how easy it would be to implement Cyrus-like
mail store which would consist of only Dovecot's index files and
"<uid>." message files. Flags and everything else is kept only in
Dovecot's index files (so it might not be the best idea to use it in
production just yet). Several hours later I ended up having it working.
The code contains:

~/cvs/m/dovecot-head/src/lib-storage/index/uiddir% wc *[ch]
  144   329  3552 uiddir-mail.c
  240   565  6206 uiddir-save.c
  532  1367 13628 uiddir-storage.c
   55   126  1712 uiddir-storage.h
  169   404  4057 uiddir-sync.c
   21    48   573 uiddir-sync.h
  103   258  2762 uiddir-transaction.c
 1264  3097 32490 total

uiddir-storage.c is still a bit too bloaty with lots of copy&pasted
code. The same with maildir and mbox:

maildir: 6022  18232 165605 total
mbox:    7859  23283 213686 total

Benchmarks show that with small mailboxes it's a lot faster than
maildir, and with larger mailboxes it's a bit faster.

I also tested it against Cyrus. I found out that fetching ENVELOPE vs.
specific headers gives quite a huge performance difference. Cyrus is
optimized for ENVELOPE while Dovecot is optimized for specific headers.
With larger mailboxes + ENVELOPE Cyrus is faster than Dovecot, while
with specific headers Dovecot is always faster. Maybe I'll some day make
Dovecot optimize for ENVELOPE also if it sees that client only wants to
use it.

BTW. Does anyone have better naming ideas than "uiddir"? I'll commit it
to CVS HEAD once I'm sure about the name. :)

u: Dovecot CVS HEAD + uiddir
m: Dovecot CVS HEAD + Maildir
c: Cyrus IMAP 2.2.13

./imaptest list=0 seed=1 secs=60 clients=10

   Logi  Stat  Sele  Fetc  Fet2  Stor  Dele Expu  Appe  Logo 
u: 23486 11822 23481 23476 33426 11659 8038 23472 11922 46952 
m: 19367 9734  19367 19356 27453  9687 6287 19347  7983 38709 
c: 5124  2553  5121  5105  7080   2559 1877  5104  3271 10228 

./imaptest list=0 dele=20 expu=20 seed=1 secs=60 msgs=10000 clients=10

   Logi Stat Sele Fetc Fet2 Stor Dele Expu Appe Logo 
u: 5291 2704 5289 5278 7448 2706 1048 1061 5283 10563
m: 4162 2103 4162 4153 5977 2124  762  853 4153 8304
c: 4302 2157 4301 4289 6021 2072  806  879 4292 8584

./imaptest list=0 dele=10 expu=10 seed=1 secs=60 msgs=100000 clients=10

   Logi Stat Sele Fetc Fet2 Stor Dele Expu Appe Logo 
u: 3211 1637 3206 3195 4639 1537  287  302 3202 6402
m: 2957 1508 2955 2944 4173 1440  279  306 2948 5894
c: 3933 1929 3933 3920 5604 2020  380  379 3923 7846

------

./imaptest list=0 dele=10 expu=10 seed=1 secs=60 msgs=100000 clients=1
   Logi Stat Sele Fetc Fet2 Stor Dele Expu Appe Logo 
u: 1529  745 1529 1528 2184  819  136  162 1528 3056 
c: 1579  813 1579 1578 2257  811  171  148 1578 3156 

./imaptest list=0 dele=10 expu=10 seed=1 secs=60 msgs=100000 clients=50
   Logi Stat Sele Fetc Fet2 Stor Dele Expu Appe Logo 
u: 3098 1552 3093 3085 4432 1520  293  334 3089 6173 
c: 3664 1765 3659 3596 5106 1802  365  364 3614 7228 

======

FETCH BODY.PEEK[HEADER.FIELDS(from to cc subject)] instead of ENVELOPE:

./imaptest list=0 dele=20 expu=20 seed=1 secs=60 msgs=10000 clients=10

   Logi Stat Sele Fetc Fet2 Stor Dele Expu Appe Logo 
u: 5777 2933 5777 5767 8215 2901 1133 1191 5767 11531 
c: 3636 1794 3635 3623 5196 1843  682  722 3627 7254 

./imaptest list=0 dele=10 expu=10 seed=1 secs=60 msgs=100000 clients=10

   Logi Stat Sele Fetc Fet2 Stor Dele Expu Appe Logo 
u: 3869 1959 3868 3861 5452 1943  364  394 3859 7717 
c: 3191 1620 3189 3184 4501 1598  315  327 3181 6362 

./imaptest list=0 dele=10 expu=10 seed=1 secs=60 msgs=100000 clients=1

   Logi Stat Sele Fetc Fet2 Stor Dele Expu Appe Logo 
u: 1912  966 1912 1910 2783  940  184  197 1911 3820 
c: 1342  696 1342 1341 1888  682  109  126 1341 2682 

./imaptest list=0 dele=10 expu=10 seed=1 secs=60 msgs=100000 clients=50

   Logi Stat Sele Fetc Fet2 Stor Dele Expu Appe Logo 
u: 4618 2320 4610 4604 6558 2266  452  455 4605 9205 
c: 2960 1472 2956 2941 4134 1425  281  312 2910 5820 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url : http://dovecot.org/pipermail/dovecot/attachments/20070329/6399e796/attachment.pgp