[Dovecot] Better to use a single large storage server or multiple smaller for mdbox?

[Timo Sirainen]

On 13.4.2012, at 15.17, Ed W wrote:

> On 13/04/2012 12:51, Timo Sirainen wrote:
>>> - Use the checksums to assist with replication speed/efficiency (dsync or custom imap commands)
>> It would be of some use with dbox index rebuilding. I don't think it would help with dsync.
> ..
>>> - File RFCs for new imap features along the "lemonde" lines which allow clients to have faster recovery from corrupted offline states...
>> Too much trouble, no one would implement it :)
> 
> I presume you have seen that cyrus is working on various distributed options?  Standardising this through imap might work if they also buy into it?

Probably more trouble than worth. I doubt anyone would want to run a cross-Dovecot/Cyrus cluster.

>>> - Storage backends where emails are redundantly stored and might not ALL be on a single server (find me the closest copy of email X) - derivations of this might be interesting for compliance archiving of messages?
>>> - Fancy key-value storage backends might use checksums as part of the key value (either for the whole or parts of the message)
>> GUID would work for these as well, without the possibility of a hash collision.
> 
> I was thinking that the win for key-value store as a backend is if you can reduce the storage requirements or do better placement of the data (mail text replicated widely, attachments stored on higher latency storage?).  Hence whilst I don't see this being a win with current options, if it were done then it would almost certainly be "per mime part", eg storing all large attachments in one place and the rest of the message somewhere else, perhaps with different redundancy levels per type
> 
> OK, this is all completely pie in the sky.  Please don't build it!  All I meant was that these are the kind of things that someone might one day desire to do and hence they would have competing requirements for what to checksum...

That can almost be done already .. the attachments are saved and accessed via a lib-fs API. It wouldn't be difficult to write a backend for some key-value databases. So with about one day's coding you could already have Dovecot save all message attachments to a key-value db, and you can configure redundancy in the db's configs.