[Dovecot] New index / mailbox API

[Timo Sirainen]

Today morning I had an idea how to make indexes more scalable with
multiple concurrent writers. As a side effect it also made the locking
issues much more simpler. So I thought I'd go and rewrite all the
indexing code, it had gotten way too ugly and difficult to understand
and maintain.

I also thought to separate the index handling completely from mailbox
handling. So there would be lib-index and lib-mailbox. lib-storage would
then tie them together.

Mailbox API still feels somewhat dirty though.. I didn't want it to
depend on lib-index and I didn't want to duplicate any lib-index
functionality in it to avoid uselessly allocating memory, so I can't
reference to mails directly with their sequence number or UID. So I'm
passing (void *id, size_t id_len) around - with maildir that would be
the filename and with mbox it would be current offset in file.

Suggestions welcome about both APIs.

Some of the changes in indexes and it's API:

- "modify log" -> "transaction log" which was the big change. All flag
updates and expunges are first written to transaction log and only when
committing it they will be moved into the main index.

The transaction writes require the file locked for only very short time,
so with lots of concurrent writers it means that several sessions could
have written their wanted changes to log file, but they're all blocking
on getting them moved into main index. When one session finally gets a
lock in the main lock file, it writes everyone's changes into it. If
lock can't be acquired in a second or so, the main index will be
rewritten into temp file and rename()d over.

This means that we can have tons of sessions reading and writing to same
shared index file and no session can block others from reading or
writing. Currently yoou could easily block changes for a very long time
with eg. FETCH 1:* BODY.PEEK[] and reading it 1 bytes/sec.

- mail_index will be a class interface, so it's implementation can be
changed at runtime. In-memory indexing is going to be a totally
different implementation this time with better memory usage. Could be
useful for optimizing some special cases as well.

- You'll get "views" into mailbox. A view will take care of keeping
sequences synchronized according to what each client session thinks they
are. Currently lib-storage does that in kind of ugly way. Hopefully this
means that we can get rid of sequence parsing in lib-storage completely
and move it into imap-only code.

- You don't directly manipulate with locks and there's no rules of in
which order you have to do something. You get views and transactions and
use them.

- You can have multiple views and transactions, so one opened index
could be used to handle multiple client sessions. I'm not sure if that
will ever be useful though :)

- Error correction will be done automatically this time. Whenever some
error is noticed, it will be fixed immediately and unless it changes
syncing state, it doesn't force client to disconnect. Also it hopefully
will be possible to change from disk-indexing to memory-indexing on the
fly in case you run out of disk space.

- I'm trying to get it NFS-safe..

- mail-cache that I recently rewrote doesn't change. The bad thing is
that it seems to be broken in some way and probably needs rewriting
anyway.. And I'm not sure if the problem is just in the code, or if the
design itself is somehow broken with it. I'm fearing that memory doesn't
somehow keep up with changes by other sessions, since we're not locking
cache file for reading..

-------------- next part --------------
A non-text attachment was scrubbed...
Name: mail-index.h
Type: text/x-c-header
Size: 8103 bytes
Desc: not available
URL: <http://dovecot.org/pipermail/dovecot/attachments/20031122/3091859e/attachment-0003.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mailbox-driver.h
Type: text/x-c-header
Size: 3102 bytes
Desc: not available
URL: <http://dovecot.org/pipermail/dovecot/attachments/20031122/3091859e/attachment-0004.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://dovecot.org/pipermail/dovecot/attachments/20031122/3091859e/attachment-0005.bin>