The biggest problem with getting v1.0 out was getting its index file code stable. v1.1 plans included doing large changes to the index file code, so it's important to get the new code stable as soon as possible. Since I've managed to stay pretty productive for the last few weeks, I've been mostly just coding the index changes. Once I'm sure that the code is again fully working I'll commit it to hg, relax a bit and then I'll again start answering mails in this list.

It looks like the code is almost working, but it's still giving failures once in a while. Below are anyway a few benchmarks for you to see how it's progressing. I haven't yet done all the optimizations I've planned, but looks like it already gives 25-30% better performance in cydir tests. The cydir benchmarks are the most important ones because cydir relies completely on index files, so the benchmarks are practically index benchmarks. And the 25-30% is mostly about CPU usage I think, reduced dovecot.index file rewrites will help with disk I/O as well.

The new index code should be pretty NFS friendly. I think there is going to be only one place where I'll need to force attribute cache flush and then probably a couple of places where data cache needs to be flushed (I'm not exactly sure about that yet, I'll have to know more about NFS data caching).

One other thing I'm hoping to implement is that the index functions would almost never fail. If anything breaks, the indexes should be moved to memory. Any corrupted data should be ignored and anything that looks good would still be used (important with cydir/dbox).

I've even managed to remove about 500 lines of code: 26 files changed, 3258 insertions(+), 3753 deletions(-) (about 2000 insert+deletions are just cut&pasting code to different files)

Append-only tests:

mmap_disable=no, fsync_disable=yes Current hg (before index rewrite), cydir:

./imaptest - append=100,0 logout=0 select=0 clients=1 secs=5 Logi Appe 100% 100% 1 2602 1/ 1 0 2671 1/ 1 0 2750 1/ 1 0 2363 1/ 1 Totals: 1 13206

Index rewrite, cydir:

Totals: Logi Appe 100% 100% 1 4251 1/ 1 0 3914 1/ 1 0 3514 1/ 1 0 2985 1/ 1 0 2538 1/ 1 Totals: 1 17202

Maildir (not much difference between rewrite/no rewrite): Logi Appe 100% 100% 1 917 1/ 1 0 498 1/ 1 0 284 1/ 1 0 273 1/ 1 0 296 1/ 1 Totals: 1 2268

mbox (+300 with index rewrite): Logi Appe 100% 100% 1 1827 1/ 1 0 1688 1/ 1 0 1811 1/ 1 0 2040 1/ 1 Totals: 1 9114

Then a bit more generic test:

Old index code, cydir:

./imaptest secs=10 clients=10 Logi List Stat Sele Fetc Fet2 Stor Dele Expu Appe Logo 100% 50% 50% 100% 100% 100% 50% 100% 100% 100% 100% 30% 5%
3417 1713 1713 3416 3403 4840 1719 1260 3403 1999 6816

New index code, cydir:

Logi List Stat Sele Fetc Fet2 Stor Dele Expu Appe Logo 100% 50% 50% 100% 100% 100% 50% 100% 100% 100% 100% 30% 5%
4686 2349 2317 4678 4675 6636 2323 1628 4675 2585 9353