On Wed, 2013-03-27 at 21:25 +0200, Timo Sirainen wrote:

The metadata plugin started with implementing the IMAP commands and kind of ignored the difficult parts of fitting it in with everything. The stuff that is in Dovecot now handles most of the difficult parts, but the IMAP commands are missing. There are still two somewhat difficult things missing:

So, here are some plans for them:

• Where/how to keep track of metadata quota to prevent abuse? Maybe some limits to number of entries, max. bytes total in entries, max. bytes per entry. Although the last one only is useful for those backends that can't handle large entry sizes.

I guess there should be a total number of metadata items and total bytes of metadata, similar as to mails. Possibly it could even share the same quotas by default.

Whenever metadata item is updated, its key is logged to dovecot.index.log among (in future version) the value's length. These can be used to update quota and minimize locking waits:

• remember current transaction log offset
• read existing modified keys' value lengths
• mailbox metadata lock
• update new keys' values, remember their lengths
• commit transaction
• mailbox metadata unlock
• scan transaction log for changes to same keys, replace old key length with latest seen key length
• increase quota with (new length - previous length)

Would be nice to get rid of the locking, but I guess it's not possible without causing a race condition (S1, S2 = session 1/2):

S1: read 10 S1: write 25 S2: read 25 S2: write 30

a) S1 committed first: old=10, new=30 -> quota +20 S1: 25-10 = +15 S2: 30-25 = +5

b) S2 committed first (which can't happen with locking): old=10, new=25 -> quota +15 S1: 25-30 = -5 S2: 30-25 = +5 [30-10=+20 would be correct..]

• Currently it's configured via mail_attribute_dict, which can e.g. point to a file in user's home directory. How to handle shared metadata in shared folders? Should userA just be accessing userB's dict file? Where are metadata stored for public folders that have no owner? Should there be two dict settings for private and public dict? (The last, or maybe all, would be a non-issue if I had already implemented my idea of having only one dict configuration where different prefixes could be mapped to completely different locations/backends/etc.)

METADATA RFC says that shared metadata is required, while private metadata isn't required. So most of the metadata is probably marked as shared, even for mailboxes without any ACLs. So for efficiency most of the metadata should be stored in each user's own metadata database rather than one huge shared database. So that's why mail_attribute_dict must be used for private and shared metadata for your user's own mailboxes. That also forces using the same for shared mailboxes.

For public mailboxes there is no owner user, so I guess there needs to be a new setting mail_attribute_public_dict. I was also considering a per-namespace mail_attribute_dict, but maybe that's not worth the trouble for now. In any case in future the idea is to allow all mail settings to be overridden by putting them inside namespace {}.