Some updates I thought of while reading through this mail..:
On Tue, 2006-10-24 at 21:31 +0300, Timo Sirainen wrote:
Of course the new mails' contents also have to be sent. This could be prioritized lower than the transaction traffic, so that each server always has very up-to-date view of the mailbox metadata, but not necessarily the contents of all the mails.
I'm not sure which is the best way to do this split of transaction data and mail contents. There are two possibilities:
Use multiple connections: high priority (requested mail contents), normal priority (transactions) and low-priority (non-requested mail contents). This won't give any priority guarantees though, unless different ports are used and admin sets up QoS. Although this won't really matter as long as there's enough bandwidth available.
Use one connection and send all data in smaller blocks. Dynamically adjust the block size to be such that it gives high enough throughput but also small enough latency that higher priority blocks can be inserted in the queue and received quickly.
If the server finds itself in a situation that it doesn't have some specific mail, it'll send a request to the replication process to fetch it ASAP from another server. The reply will then take the highest priority in the queue.
If the other server is gone, there's no way to get the mail. In this case the mail needs to be treated as expunged, and in the next sync it'll get a new UID.
Now, when exactly should these replication-writer processes be created then? If all users have the same UNIX user ID, then there could be a spool of processes always running and doing the writing. Although this worries me a bit since the same process is handling different users' data.
More secure way, and the only way if users have different UNIX UIDs, is to launch a separate writer process for each user.
Note that this exact same problem comes if LMTP server is implemented.