[Dovecot] Dovecot METADATA support

[Timo Sirainen]

On Wed, 2013-03-27 at 21:25 +0200, Timo Sirainen wrote:

> The metadata plugin started with implementing the IMAP commands and
> kind of ignored the difficult parts of fitting it in with everything.
> The stuff that is in Dovecot now handles most of the difficult parts,
> but the IMAP commands are missing. There are still two somewhat
> difficult things missing:

So, here are some plans for them:

>  * Where/how to keep track of metadata quota to prevent abuse? Maybe
> some limits to number of entries, max. bytes total in entries, max.
> bytes per entry. Although the last one only is useful for those
> backends that can't handle large entry sizes.

I guess there should be a total number of metadata items and total bytes
of metadata, similar as to mails. Possibly it could even share the same
quotas by default.

Whenever metadata item is updated, its key is logged to
dovecot.index.log among (in future version) the value's length. These
can be used to update quota and minimize locking waits:

 - remember current transaction log offset
 - read existing modified keys' value lengths
 - mailbox metadata lock
 - update new keys' values, remember their lengths
 - commit transaction
 - mailbox metadata unlock
 - scan transaction log for changes to same keys, replace old key length
with latest seen key length
 - increase quota with (new length - previous length)

Would be nice to get rid of the locking, but I guess it's not possible
without causing a race condition (S1, S2 = session 1/2):

S1: read 10
S1: write 25
S2: read 25
S2: write 30

a) S1 committed first:
old=10, new=30 -> quota +20
S1: 25-10 = +15
S2: 30-25 = +5

b) S2 committed first (which can't happen with locking):
old=10, new=25 -> quota +15
S1: 25-30 = -5
S2: 30-25 = +5 [30-10=+20 would be correct..]

>  * Currently it's configured via mail_attribute_dict, which can e.g.
> point to a file in user's home directory. How to handle shared
> metadata in shared folders? Should userA just be accessing userB's
> dict file? Where are metadata stored for public folders that have no
> owner? Should there be two dict settings for private and public dict?
> (The last, or maybe all, would be a non-issue if I had already
> implemented my idea of having only one dict configuration where
> different prefixes could be mapped to completely different
> locations/backends/etc.)

METADATA RFC says that shared metadata is required, while private
metadata isn't required. So most of the metadata is probably marked as
shared, even for mailboxes without any ACLs. So for efficiency most of
the metadata should be stored in each user's own metadata database
rather than one huge shared database. So that's why mail_attribute_dict
must be used for private and shared metadata for your user's own
mailboxes. That also forces using the same for shared mailboxes.

For public mailboxes there is no owner user, so I guess there needs to
be a new setting mail_attribute_public_dict. I was also considering a
per-namespace mail_attribute_dict, but maybe that's not worth the
trouble for now. In any case in future the idea is to allow all mail
settings to be overridden by putting them inside namespace {}.