On 03/07/2011 07:20, Peter Dolding wrote:
Now what I need to be able to deal with the problem.
Have you considered a new Dovecot storage backend? I have plotted some designs on a napkin a few times.. Consider some kind of storage server with "eventually consistent" replication capabilities. This could be used for the metadata storage for all the emails (ie FROM, TO, DATE, SUBJECT and all the other non body parts you might search on)
Your replication engine can now work in conjunction with Dovecot to sync changes between servers as quickly as possible, eg if desired implement a two phase commit when the LDA delivers new emails, so that all storage servers confirm they have received the new email. You could if you wish implement quorum support (ie some server which was offline for some mail deliveries proxies requests to another storage block until it's caught up syncing)
You may or may not store the message bodies and attachments with the mail metadata. I can see performance arguments depending on what operations you do most commonly. A theoretically (but probably horrible in practice) idea might be to consider "DB query latency" vs transfer speed. The metadata needs to cover 99% of common searches and deliver results quickly - as such it needs to be "near" Dovecot and cover the main message headers, and perhaps also message body structure (list of parts, etc). Next tier down is probably satisfying full text searches of message bodies and supplying body parts (where the most common queries might be either just text/html parts (smart clients) or the entire body (common clients). Some kind of compressed blocking format for message bodies is probably most optimal and storing larger attachments separately would be an interesting way to increase cache hit ratios.
I don't know whether Timo is interested in working such a project, but I for one would be interested to sponsor some work on robust async replication, perhaps there is some crossover with synchronous replication that you desire?
I think this is an interesting area to develop. Cyrus has done some work on this stuff - I haven't followed it, but would be interesting to see what they have done?
Good luck
Ed W