[Dovecot] Replication plans

[Troy Benjegerdes]

> > This increases communication and locking significantly. The locking alone 
> > will likely be a choke point. 
> 
> My plan would require the locking only when the mailbox is being updated
> and the global lock isn't already owned by the server. If you want to
> avoid different servers from constantly stealing the lock from each
> others, use different ways to make sure that the mailbox normally isn't
> modified from more than one server.
> 
> I don't think this will be a big problem even if multiple servers are
> modifying the same mailbox, but it depends entirely on the extra latency
> caused by the global locking. I don't know what the latency will be
> until it can be tested, but I don't think it should be much more than
> what a simple ping would give over the same network.

Best case, when all the nodes, and the network is up, locking latency
shouldn't be much longer than say twice the RTT. But what really
matters, and causes all the nasty bugs that even single-master
replication systems have to deal with is the *worst case* latency. So
everything is going along fine, and then due to a surge in incoming
spam, one of your switches starts dropping 2% of the packets, and the
server holding a lock starts taking 50ms instead of 1ms to respond to an
incoming packet. 

Now your previous lock latency of 1ms could easily extend into seconds if
a couple of responses to lock requests don't get through. And your 16
node imap cluster is now 8 times slower than a single server, instead of
8 times faster ;)

The nasty part about this for imap is that we can't ever have a UID be
handed out without *confirming* that it's been replicated to another
server before sending out the packet. Otherwise you can get in the
situation where node A sends out a new UID to a client out it's public
NIC card, while in the meantime, it's internal NIC melted so the update
never got propagated, so node B,C, and D  decides "ooops, node A is
dead, we are stealing his lock", and B takes over the lock and allocates
the same UID to a different message, and now the CEO didn't get that
notice from the SEC to save all his emails.


Once you decide you want replication, you pretty much have to go all the
way to synchronous replication, and now you have a learning curve and
complexity issue that's going to be there whether it's dovecot
replication, or a cluster filesystem that's doing the dirty work for
you.


-- 
--------------------------------------------------------------------------
Troy Benjegerdes                'da hozer'                hozer at hozed.org  

Somone asked me why I work on this free (http://www.fsf.org/philosophy/)
software stuff and not get a real job. Charles Shultz had the best answer:

"Why do musicians compose symphonies and poets write poems? They do it
because life wouldn't have any meaning for them if they didn't. That's why
I draw cartoons. It's my life." -- Charles Shultz