[Dovecot] key -> object mailstore

[Damien Churchill]

On 17 September 2012 13:57, Timo Sirainen <tss at iki.fi> wrote:
> On 14.9.2012, at 18.16, Damien Churchill wrote:
>
>> On 14 September 2012 15:59, Timo Sirainen <tss at iki.fi> wrote:
>>> On 14.9.2012, at 17.46, Damien Churchill wrote:
>>>
>>>> I was wondering what would be entailed in modifying dovecot 2.2 to
>>>> support storing mail in an object store. I've seen a few mails dotted
>>>> around in the ML history about supporting such a thing and seen it's
>>>> basically dependant upon some changes in lib-storage to support
>>>> writing messages without locking. Is this still the case?
>>>
>>> I've a whole new design for it and I was planning on implementing it for v2.2. Do you want to help coding it? :) Which storage would you want to use?
>>>
>>
>> That's good to hear :) I've been evaluating RADOS as an object store,
>> which is similar to S3. Although any distributed storage would be
>> great. I'd be more than happy to help code it!
>
> I think I'll first have to get started with it to see if there are some parts that are easy to give to you. First I'll at least need to do some refactoring to dbox code and lib-fs code. I'm planning on making the generic parts of it be part of Dovecot releases, but I haven't yet fully decided which backends should be..
>
>>> The generic idea is:
>>> - only one server accesses one user simultaneously
>>> - index files are copied from object storage to local filesystem and accessed there, once in a while uploaded back to object storage
>>> - if user is accessed from two servers because of some bug/split brain/something, the changes are merged using dsync
>>> - support high latency: asynchronous reads/writes. prefetch mail bodies.
>>
>> I'm assuming that the director would be used in order to distribute
>> connections to the same server, so it's only within a local instance
>> of dovecot you'd need to be aware of what currently has a connection
>> open for that user?
>
> Right. Probably some new process that can do the work of downloading/uploading/deleting index files as needed. That's actually a clearly separate task that you could do? :)
>

Sounds good! I'll spend some time digging through the source code
getting familiar.

>> How are you planning on handling the situation where say node X dies
>> and hasn't uploaded the latest index file? Would that result in
>> missing messages from the mailbox when accessed by another node, or is
>> the local index intended to be more of a write-through cache?
>
> No messages get ever lost. Recent flag changes and expunges may get lost, at least until the original node comes back up and dsync merges the changes. Idea was that when downloading index a flag on the object storage is set for the user that it's being accessed, and removed after the user is disconnected and index is uploaded back. If index downloader already sees that the flag is set it will run some kind of a recovery process to find any messages that were uploaded but not indexed. (The message bodies are always immediately uploaded to object storage.)

Part of me thinks making this configurable might be a good idea
depending upon what the installation is trying to achieve. Since the
recovery process will need to be implemented regardless allowing the
user to configure dovecot to perform a write to both the local and
object index when a flag or something else is modified.

Another thought occurs to me, when using the LDA how will that be able
to update the index upon delivery of a new message if another node is
currently accessing the mailbox?