Today morning I had an idea how to make indexes more scalable with multiple concurrent writers. As a side effect it also made the locking issues much more simpler. So I thought I'd go and rewrite all the indexing code, it had gotten way too ugly and difficult to understand and maintain.
I also thought to separate the index handling completely from mailbox handling. So there would be lib-index and lib-mailbox. lib-storage would then tie them together.
Mailbox API still feels somewhat dirty though.. I didn't want it to depend on lib-index and I didn't want to duplicate any lib-index functionality in it to avoid uselessly allocating memory, so I can't reference to mails directly with their sequence number or UID. So I'm passing (void *id, size_t id_len) around - with maildir that would be the filename and with mbox it would be current offset in file.
Suggestions welcome about both APIs.
Some of the changes in indexes and it's API:
- "modify log" -> "transaction log" which was the big change. All flag updates and expunges are first written to transaction log and only when committing it they will be moved into the main index.
The transaction writes require the file locked for only very short time, so with lots of concurrent writers it means that several sessions could have written their wanted changes to log file, but they're all blocking on getting them moved into main index. When one session finally gets a lock in the main lock file, it writes everyone's changes into it. If lock can't be acquired in a second or so, the main index will be rewritten into temp file and rename()d over.
This means that we can have tons of sessions reading and writing to same shared index file and no session can block others from reading or writing. Currently yoou could easily block changes for a very long time with eg. FETCH 1:* BODY.PEEK[] and reading it 1 bytes/sec.
mail_index will be a class interface, so it's implementation can be changed at runtime. In-memory indexing is going to be a totally different implementation this time with better memory usage. Could be useful for optimizing some special cases as well.
You'll get "views" into mailbox. A view will take care of keeping sequences synchronized according to what each client session thinks they are. Currently lib-storage does that in kind of ugly way. Hopefully this means that we can get rid of sequence parsing in lib-storage completely and move it into imap-only code.
You don't directly manipulate with locks and there's no rules of in which order you have to do something. You get views and transactions and use them.
You can have multiple views and transactions, so one opened index could be used to handle multiple client sessions. I'm not sure if that will ever be useful though :)
Error correction will be done automatically this time. Whenever some error is noticed, it will be fixed immediately and unless it changes syncing state, it doesn't force client to disconnect. Also it hopefully will be possible to change from disk-indexing to memory-indexing on the fly in case you run out of disk space.
I'm trying to get it NFS-safe..
mail-cache that I recently rewrote doesn't change. The bad thing is that it seems to be broken in some way and probably needs rewriting anyway.. And I'm not sure if the problem is just in the code, or if the design itself is somehow broken with it. I'm fearing that memory doesn't somehow keep up with changes by other sessions, since we're not locking cache file for reading..