[Dovecot] Integrating Dovecot with Amazon Web Services

[Timo Sirainen]

On 28.6.2012, at 20.55, Gary Mort wrote:

>> The indexes have to be in primary storage.
>> 
> True, but the data they are based on I'm assuming does not include the full
> email message, just a few key pieces:
> uniqueid, subject, from, to, etc.
> 
> For an always running server, the indexes are always up to date in primary.
> 
> For a server starting up with no index data, it will need to rebuild the
> index information[or for a second server running when new email has been
> delivered].
> As such, rather then download every single email message just for a few
> bits of key info, I can run a re-index process to pull just the meta
> information and grab the data from there.

With sdbox you can't lose index files without also losing all message flags. And in general sdbox assumes that indexes are always up to date.

>>> When a client attempts to retrieve an email message, Dovecot would check
>>> primary storage as it does now, if the message is not found than it will
>>> retrieve it from the alternate storage system AND store a copy in the
>>> primary storage.
>> 
>> I think the storing wouldn't be very useful. Most clients download the
>> message once. There's no reason to cache it if it doesn't get downloaded
>> again. The way it should work that new mails are immediately delivered to
>> both primary and alt storage.
>> 
>> 
> I've got tons of space - so I don't mind having 750MB or so for primary
> email message storage.   If I can track how many times a message was
> actually read, over time I can get an idea of how I use it and setup the
> primary storage purge rules accordingly.

I'd be interested in knowing what those statistics will end up looking like. My guess is that it's not worth coding such feature, but of course some real world data would be better than my guesses :)

>>> Secondly, I'd like to replace the Mysql database usage with a simpleDB
>>> database.  While simpleDB lacks much of MySQL's sophistication, it
>> doesn't
>>> seem that Dovecot is really using any of that, so simpleDB can be
>>> functionally equivalent.
>> 
>> Dovecot will probably get Redis and/or memcache backend for passdb+userdb.
>> If simpledb is similar key-value database I guess the same code could be
>> used partially.
>> 
>> 
> simpleDB is more like SQLLITE:
..
> You query the data like an SQL table:
> http://docs.amazonwebservices.com/AmazonSimpleDB/latest/DeveloperGuide/UsingSelect.html

OK, so that would mean implementing lib-sql driver for SimpleDB and use sql passdb/userdb.