[Dovecot] Index file rewrite status

[Timo Sirainen]

The biggest problem with getting v1.0 out was getting its index file
code stable. v1.1 plans included doing large changes to the index file
code, so it's important to get the new code stable as soon as possible.
Since I've managed to stay pretty productive for the last few weeks,
I've been mostly just coding the index changes. Once I'm sure that the
code is again fully working I'll commit it to hg, relax a bit and then
I'll again start answering mails in this list.

It looks like the code is almost working, but it's still giving failures
once in a while. Below are anyway a few benchmarks for you to see how
it's progressing. I haven't yet done all the optimizations I've planned,
but looks like it already gives 25-30% better performance in cydir
tests. The cydir benchmarks are the most important ones because cydir
relies completely on index files, so the benchmarks are practically
index benchmarks. And the 25-30% is mostly about CPU usage I think,
reduced dovecot.index file rewrites will help with disk I/O as well.

The new index code should be pretty NFS friendly. I think there is going
to be only one place where I'll need to force attribute cache flush and
then probably a couple of places where data cache needs to be flushed
(I'm not exactly sure about that yet, I'll have to know more about NFS
data caching).

One other thing I'm hoping to implement is that the index functions
would almost never fail. If anything breaks, the indexes should be moved
to memory. Any corrupted data should be ignored and anything that looks
good would still be used (important with cydir/dbox).

I've even managed to remove about 500 lines of code:
 26 files changed, 3258 insertions(+), 3753 deletions(-)
(about 2000 insert+deletions are just cut&pasting code to different
files)

Append-only tests:

mmap_disable=no, fsync_disable=yes
Current hg (before index rewrite), cydir:

./imaptest - append=100,0 logout=0 select=0 clients=1 secs=5
Logi Appe 
100% 100% 
   1 2602   1/  1
   0 2671   1/  1
   0 2750   1/  1
   0 2363   1/  1
Totals:
   1 13206 

Index rewrite, cydir:

Totals:
Logi Appe 
100% 100% 
   1 4251   1/  1
   0 3914   1/  1
   0 3514   1/  1
   0 2985   1/  1
   0 2538   1/  1
Totals:
   1 17202 

Maildir (not much difference between rewrite/no rewrite):
Logi Appe 
100% 100% 
   1  917   1/  1
   0  498   1/  1
   0  284   1/  1
   0  273   1/  1
   0  296   1/  1
Totals:
   1 2268 

mbox (+300 with index rewrite):
Logi Appe 
100% 100% 
   1 1827   1/  1
   0 1688   1/  1
   0 1811   1/  1
   0 2040   1/  1
Totals:
   1 9114 

Then a bit more generic test:

Old index code, cydir:

./imaptest secs=10 clients=10
Logi List Stat Sele Fetc Fet2 Stor Dele Expu Appe Logo 
100%  50%  50% 100% 100% 100%  50% 100% 100% 100% 100% 
                          30%                  5%      
3417 1713 1713 3416 3403 4840 1719 1260 3403 1999 6816 

New index code, cydir:

Logi List Stat Sele Fetc Fet2 Stor Dele Expu Appe Logo 
100%  50%  50% 100% 100% 100%  50% 100% 100% 100% 100% 
                          30%                  5%      
4686 2349 2317 4678 4675 6636 2323 1628 4675 2585 9353 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url : http://dovecot.org/pipermail/dovecot/attachments/20070530/b3b0c8a0/attachment.bin