[Dovecot] 1.0-test25 with new cache file code

[Timo Sirainen]

http://dovecot.org/test/

Pretty much the only change since test24 is a redesign of how cache 
file writing works. Reading works as before - without locks.

Before (in 1.0-tests, 0.99 was worse) we used to just try to lock it. 
If it failed, we just didn't update cache. If it didn't fail, the lock 
could have been kept a long time. For example if a user was downloading 
a 10MB mail, the lock could have been kept the whole time and no-one 
else could have modified the cache.

The old behaviour wasn't actually too bad, as normally it didn't matter 
if something couldn't be written to cache. It would have been the next 
time, all it did was cost some I/O.

The new behaviour is more reliable as the locks are kept only a very 
short time. It works by first writing the updates only into internal 
32kB buffer. Once the buffer gets full, or the transaction completes, 
it's written to disk. When buffer gets full, space is reserved from the 
cache file for the current transaction. The amount of space is more 
than is needed for writing the internal buffer, so we don't have to 
lock the cache file every time the buffer gets full (and so we don't 
need a huge buffer).

Once transaction completes, we lock the cache again and try to free the 
extra reserved space we have left. If multiple processes weren't 
updating the cache file at the same time, the freeing should be as 
simple as decreasing "used_file_size" field in header. Alternatively 
we'll update a linked list of "holes" in the file. When reserving 
space, we first try to use those holes instead of appending to end of 
file.

Pointers to new cache records are also written to transaction log at 
the end of cache transaction. They'll get written to index at next sync 
after which they become visible to other processes. Each cache record 
contains a pointer to previous record, so there can be multiple cache 
records for a single message.

The bad thing with the new logic is that nothing prevents two processes 
from writing the same cache records and so using twice as much space as 
necessary. But I don't think it's much of a problem. They'll go away at 
next cache compression.

These changes also make it quite simple to make cache file work with 
shared filesystems. I'd just need to implement some kind of internal 
caching buffer which remembers what parts of file are read into memory.

These changes also make it possible to easily and reliably store X-UIDL 
headers into cache file, where the POP3 UIDL command could quickly get 
them. Not in this release yet, though..
-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 186 bytes
Desc: This is a digitally signed message part
URL: <http://dovecot.org/pipermail/dovecot/attachments/20040705/4a6bee66/attachment-0001.bin>