[Dovecot] 1.0-test23 and caching decisions

[Timo Sirainen]

http://dovecot.org/test/

mbox fixes mostly. Began some other things, but none of them actually
work yet:

 - Use X-UIDL header in mboxes for POP3 UIDL list
 - Cache file compression
 - Some cleverness for cache file for figuring out what to cache

Well, I might as well explain the last one now.

Users can be divided to three groups:

1. Most users will use only a single IMAP client which caches everything
locally. For these users it's quite pointless to do any kind of caching
as it only wastes disk space. That might also mean more disk I/O.

2. Some users use multiple IMAP clients which cache everything locally.
These could benefit from caching until all clients have fetched the
data. After that it's useless.

3. Some clients don't do permanent local caching at all. For example
Pine and webmails. These clients would benefit from caching everything.
Some locally caching clients might also access some data from server
again, such as when searching messages. They could benefit from caching
only these fields.

After thinking about these a while, I figured out that people who care
about performance most will be using the upcoming Dovecot LDA anyway
which updates the indexes/cache immediately. In that case even the first
user group would benefit from caching the same way as second group. LDA
reads the mail anyway, so it might as well extract some information
about it and store them into cache.

So, group 1. and 2. could be optimally implemented by keeping things
cached only for a while. I thought a week would be good. When cache file
is compressed, everything older than week will be dropped.

But how to figure out if user is in group 3? One quite easy rule would
be to see if client is accessing messages older than a week. But with
only that rule we might have already dropped useful cached data. It's
not very nice if we have to read and cache it twice.

Most locally caching clients always fetch new messages (all but body)
when they see them. They fetch them in ascending order. Noncaching
clients might fetch messages in pretty much any order, as they usually
don't fetch everything they can, only what's visible in screen. Some
will use server side sorting/threading which also makes messages to be
fetched in random order. Second rule would then be that if a session
doesn't fetch messages in ascending order, the fetched field type will
be permanently cached.

So, we have three caching decisions:

1. Don't cache: Clients have never wanted the field
2. Cache temporarily: Clients want this only once
3. Cache permanently: Clients want this more than once

Different mailboxes have different decisions. Different fields have
different decisions.

There are some problems, such as if a client accesses message older than
a week, we can't know if user just started using a new client which is
just filling it's local cache for the first time. Or it might be a
client user hasn't just used for over a week. In these cases we
shouldn't have marked the field to be permanently cached. User might
also switch clients from non-caching to caching.

So we should re-evaluate our caching decisions from time to time. This
is done by checking the above rules constantly and marking when was the
last time the decision was right. If decision hasn't matched for two
months, it's changed. I picked two months because people go to at least
one month vacations where they might still be reading mails, but with
different clients.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://dovecot.org/pipermail/dovecot/attachments/20040630/ee201507/attachment-0001.bin>