Mon, 04 Jul 2011 10:46:31 Ed W On 03/07/2011 07:20, Peter Dolding wrote:

...

Now what I need to be able to deal with the problem.

Have you considered a new Dovecot storage backend? I have plotted some designs on a napkin a few times.. Consider some kind of storage server with "eventually consistent" replication capabilities. This could be used for the metadata storage for all the emails (ie FROM, TO, DATE, SUBJECT and all the other non body parts you might search on)

Remember I am new to the Dovecot source code. How to code a Dovecot Storage backend it where I might have to start.

Really I don't care if the servers are ever fully consistent other than the fact they both contain the same emails. Of course read status shared would be nice. If that status information is out so be it.

"eventually consistent" is a deadly thing to try to aim for.

Its a simple fact of what needs to be backed up. If a user is at one physical location all the time and connecting to the same server all the time they will not ever see that the 2 servers are not 100 percent synced.

For my usage cases lot of times 100 percent synced will be more wasted effort. Roughly synced will be more than suitable.

Of course what I am talking about can be used as an foundation to get data from point a to point b.

Your replication engine can now work in conjunction with Dovecot to sync changes between servers as quickly as possible, eg if desired implement a two phase commit when the LDA delivers new emails, so that all storage servers confirm they have received the new email. You could if you wish implement quorum support (ie some server which was offline for some mail deliveries proxies requests to another storage block until it's caught up syncing)

You are missing something here. My sync's due to issues maybe chaos. So both servers may have been running split from each other both received emails the other has not had users accessing them. Both with users in using the web and local email clients. While all that is going on the servers have to sync.

This is very much the worst case. I am sane. I am prepared to give up the users ability to change client programs between servers to make it be able to work. So IMAP unique identifies not be replicated ever. Basically any identifier that cannot be based of a server id + a unique number for server be only unique to the server its on.

Basically I cannot ensure quickly and most of the time I don't really need that. As long as the email without breakage is not lost in the server for greater than 15 min without reaching person wanting to read it that would be fine for normal operations. This is still faster than exchange working with pop email accounts.

Basically Ed W. I am saying this is the worst I can get away with in a working office.

I have gone through the protocals I would most likely ever need to use from a business point of view.

pop3 I know basically does not really give a stuff if what is hiding behind is multi master or single master. User might receive a few extra emails if they change between servers. Nothing system killing. As long as the user gets the emails in the end that is the important bit if they a copy for each server bad luck at least they got the email. Too many copies is not a issue from business point of view not getting the copy in the first place is an issue.

smtp messages directly from multi locations outgoing does not give a stuff.

imap4 who ever invented this protocol for what I want todo I fell like strangling. The id system completely sends you to hell. Business point of view this might be a issue if uses change between servers due to having to download everything again.

activesync/Z-push Ok nice. ID are 64 char in size with no defined contents. No ascending order no trouble basically. So nothing stopping me doing server_id:then unique number. So that is most mobile devices covered. Message might disappear when moving between not totally synced servers. Message will catch up as syncs do. So from business usage annoying but issue is not long lasting. Ie custom backend on Z-push and mobile devices will work mostly fine with chaos between the storage servers.

MAPI is 64 chars to 512 chars. for ID's. Again nothing in protocol stopping server_id:then unique number.

Web applications much of a muchness. Since either they will connect to the same server as the http server at each location so be protected or can be hooked onto a new protocol to know they are connecting to a multi server back end and detect if there are issues going on. Mostly as bad as activesync if you change between hosting locations email syncs might not have caught up with you so a message might disappear temp.

So only imap4 is requiring syncing so ids are in order. Question does everyone need imap4 server locations to be interchangeable. I know I don't. Since most people out the office either use web mail or activesync. Only in office will be using imap4.

Now I cannot see any fast sane way for imap4 for other than make it id only unique to each server and maybe create like imap 5 for multi server with the option to run locally as a imap5 to imap4 converter. Reason you ask server when it synced with X server and you find out that is older than the copies you have you don't replace. So the client is hiding server to server sync issues.

Sometimes the best solution is look at everything and see what is limiting you and deprecate it. If we don't need id aligned we don't need real-time connection. We don't need as reliable of an network. So failures are less.

I guess everyone who has tried to sync has attempted to battle the imap protocal. I am mostly the first strange person who has said stuff syncing imap and just go straight past its defect and make the storage work. Simple fault in imap design make the complete back end replication a pain in but when it should be so simple. I have decided of someone else wants imap to be synced let that be their problem.

Simple each location stores what they receive. and forwards to all other locations as those locations come on line and the other locations do the reverse. I like the idea of natural event syncing. That the items pure naturally sync with each other. No reprocessing to force syncing due to least amount conflicting data created. Possible for pop MAPI and activesync/Z-push todo natural syncing. IMAP the evil thing just does not want to play ball with natural syncing and it wants to set up events of conflicting data.

IMAP might force a real-time synced server in some cases. A master server handing out IMAP id numbers for new messages is one possibility so removing the need change id's on the mail after it is stored. If my web interfaces are using something other than IMAP I could possibly say use webmail until fixed.

Worst issues I can see is message read status and other times like that. Where user might have marked a message unread at X on one server and at Y time marked it back read on a different server. But these are more niggles. Better to have the users annoyed by minor niggles and working than unable to work because its broken.

Dependable replications the more parts you use the more areas you have for failure.

So far my research on what is possible seams to be coming out to reasonable.

Hopefully once I have this idea fully ironed out I can start getting into code. I rarely code. But this is a true case I have a problem so I have motivation.

Peter Dolding