[Dovecot] Replication plans

[Francisco Reyes]

Timo Sirainen writes:

>> Why not go with a pure log replication scheme?
>> this way you basically have 3 processes.
>> 
>> 1- The normal, currently existing programs. Add logs to the process
>> 2- A Master replication process which listens for clients requesting for 
>> info.
>> 3- The slave processes that request infomation and write it to the slave 
>> machines.
>> 
>> With this approach you can basically break it down into logical units of 
>> code which can be tested and debugged. Also helps when you need to worry 
>> about security and the level at which each component needs to work.
> 
> I'm not completely sure what you mean by these. Basically the same as
> what I said, except just have imap/deliver simply send the changes
> without any waiting?

I am basically suggesting to log all the changes to a log(s) and have a 
separate program handle passing on the information to the slaves.
  
> But isn't the point of the master/slave that the slave would switch on
> if the master dies?

Yes.

> If you switch slave to be the new master, it doesn't
> matter if the logs were written to master's disk. Sure the message could
> come back when the master is again brought back

I was thinking that once a master dies, next it it comes back it would be a 
slave and no longer master. This would, unfortunately imply, that any 
transactions that were not copied over would be lost.

> I don't understand what you mean. Sure the logs could timeout at some
> point (or shouldn't there be some size limits anyway?), but you'd still
> need to keep track of what different slaves have seen.

I was thinking that it would be the slaves job to ask for data.
Example pseudo transactions.

Master gets transactions 1 through 100
Slave(s) start from scratch and ask from transactions starting at 1.
Say one slave, let's call it A, dies at transaction 50 and  another slave, 
say B, continues and goes all the way to 100.

More transactions come and now we are up to 150.
Slave B will ask for anything after 100.
When Slave A comes back it would ask for anything after 50.

The master simply gets a request for transactions after a given number so it 
doesn't need to keep track the status of each slave.

> possible. It would be easier to implement much simpler master-slave
> replication, but in error conditions that would almost guarantee that
> some messages get lost. I want to avoid that.


If you think Synchronous replication is doable.. go for it. :-)
it is a tradeoff of reliability vs speed.
Specially as the number of slaves grow the communication will grow.

> Sure. I wasn't planning on implementing multi-slave or multi-master
> before the master/slave was fully working and stress testing showing
> that no mails get lost even if master is killed every few seconds (and
> each crash causing master/slave to switch roles randomly).

Great idea.