[Dovecot] Integrating Dovecot with Amazon Web Services

[Timo Sirainen]

On 28.6.2012, at 17.43, Gary Mort wrote:

> http://wiki2.dovecot.org/MailboxFormat/dbox
> 
> To make life easy, I'll stick with just single-dbox as a start, however
> multi-dbox would be doable.
> 
> With dbox, the only thing that I need to change is the alternate storage
> model:
> "An upshot of the way alternate storage works is that any given storage
> file (mailboxes/<folder>/dbox-Mails/u.* (sdbox) or storage/m.* (mdbox)) can
> only appear *either* in the primary storage area *or* the alternate storage
> area but not both — if the corresponding file appears in both areas then
> there is an inconsistency."

Whoever wrote that wasn't exactly correct (or clear). There's no problem having the same file in both primary and alt storage. Only if the files are different there's a problem, but that shouldn't happen..

> First I want to add AWS S3 as a storage option for alternate storage.
> 
> Then instead of the above model, the new model would be that email is
> always stored in alternate storage, and may be in primary storage.  So,
> when mail comes in, I'd have Dovecot save the email to the alternate
> storage S3 bucket and update the indexs and other information[ideally, for
> convenience purposes, a few bits of relevant indexing information can be
> stored as metadata in the S3 object  - sufficient so that instead of
> retrieving the entire S3 object, just the meta data can be pulled to build
> indexes.

The indexes have to be in primary storage.

> When a client attempts to retrieve an email message, Dovecot would check
> primary storage as it does now, if the message is not found than it will
> retrieve it from the alternate storage system AND store a copy in the
> primary storage.

I think the storing wouldn't be very useful. Most clients download the message once. There's no reason to cache it if it doesn't get downloaded again. The way it should work that new mails are immediately delivered to both primary and alt storage.

> Secondly, I'd like to replace the Mysql database usage with a simpleDB
> database.  While simpleDB lacks much of MySQL's sophistication, it doesn't
> seem that Dovecot is really using any of that, so simpleDB can be
> functionally equivalent.

Dovecot will probably get Redis and/or memcache backend for passdb+userdb. If simpledb is similar key-value database I guess the same code could be used partially.