[Dovecot] A new director service in v2.0 for NFS installations

[Timo Sirainen]

On 19.5.2010, at 16.16, luben karavelov wrote:

> On Wed, 19 May 2010 10:51:06 +0200, Timo Sirainen <tss at iki.fi> wrote:
>> The company here in Italy didn't really like such idea, so I thought
> about
>> making it more transparent and simpler to manage. The result is a new
>> "director" service, which does basically the same thing, except without
> SQL
>> database. The idea is that your load balancer can redirect connections
> to
>> one or more Dovecot proxies, which internally then figure out where the
>> user should go. So the proxies act kind of like a secondary load
> balancer
>> layer.
> 
> As I understand, the first load balancer is just IP balancer, not
> POP3/IMAP balancer, isn't it?

Right.

> I have implemented similar scheme here with imap/pop3 proxy (nginx) in 
> front of dovecot servers. What i have found to work best (for my
> conditions)
> as hashing scheme  is some sort of weighted constant hash. 

I guess you meant consistent hash? Yeah, I thought about that too first but simpler hash seemed .. simpler. :)

> In this way you do not need to synchronize a state between balancers 
> and proxies. If you add or remove servers very few clients get 
> reallocated - num active clients/num servers.
..
> This scheme has some disadvantages also - on certain circumstances, 
> different sessions to one mailbox could be handled by different
> servers in parallel.

That's the main thing I wanted to prevent with the director service, so I don't think consistent hashing would have made implementing it easier. Although it might have helped make the caching work a bit better when servers were added/removed.

> So my choice was to trade correctness (no parallel sessions to 
> different servers) for simplicity (no state synchronization between 
> servers).

I would have liked to avoid the state sync too, but I like more the idea of having it work 100% perfectly in all conditions. :)

> Also, I have a question. Your implementation, what kind of sessions does 
> it balance? I suppose imap/pop3. Is there a plan for similar redirecting
> of LMTP connections based on delivery address?

It's a pretty generic service. Currently imap and pop3 use it, but it would be pretty easy to make LMTP proxy support it too. The main difference is that for imap/pop3 it needs to emulate being an auth socket, while for lmtp it would need to emulate being a userdb socket. I'd guess it would need less than 50 lines of code total. I guess I should do it before v2.0.0.