[Dovecot] A new director service in v2.0 for NFS installations

[luben karavelov]

On Wed, 19 May 2010 10:51:06 +0200, Timo Sirainen <tss at iki.fi> wrote:
> The company here in Italy didn't really like such idea, so I thought
about
> making it more transparent and simpler to manage. The result is a new
> "director" service, which does basically the same thing, except without
SQL
> database. The idea is that your load balancer can redirect connections
to
> one or more Dovecot proxies, which internally then figure out where the
> user should go. So the proxies act kind of like a secondary load
balancer
> layer.

As I understand, the first load balancer is just IP balancer, not
POP3/IMAP balancer, isn't it?
 
> When a connection from a newly seen user arrives, it gets assigned to a
> mail server according to a function:
> 
>   host = vhosts[ md5(username) mod vhosts_count ]
> 
> This way all of the proxies assign the same user to the same host
without
> having to talk to each others. The vhosts[] is basically an array of
hosts,
> except each host is initially listed there 100 times (vhost count=100).
> This vhost count can then be increased or decreased as necessary to
change
> the host's load, probably automatically in future.
> 
> The problem is then of course that if (v)hosts are added or removed, the
> above function will return a different host than was previously used for
> the same user. That's why there is also an in-memory database that keeps
> track of username -> (hostname, timestamp) mappings. Every new
connection
> from user refreshes the timestamp. Also existing connections refresh the
> timestamp every n minutes. Once all connections are gone, the timestamp
> expires and the user is removed from database.
> 

I have implemented similar scheme here with imap/pop3 proxy (nginx) in 
front of dovecot servers. What i have found to work best (for my
conditions)
as hashing scheme  is some sort of weighted constant hash. 
Here is the algorithm I use:

On init, server add or server remove you initialize a ring:

1. For every server:
   - seed the random number generator with the crc32(IP of the server)
   - get N random numbers (where N = server weight) and put them in an 
     array. Put randon_number => IP in another map/hash structure.
2. Sort the array. This is the ring.

For every redirect request:

1. get crc32 number of the mailbox
2. traverse the ring until you  find a number that is bigger than 
   the crc32 number and was not yet visited. 
3. mark that number as visited.
4. lookup if it is already marked dead. If it was marked goto 2.
5. lookup the number in the map/hash and you find the IP of the server. 
6. redirect the client to that server
7. If that server is not responding, you mark it as dead and goto 2.

In this way you do not need to synchronize a state between balancers 
and proxies. If you add or remove servers very few clients get 
reallocated - num active clients/num servers. If one server is not 
responding, the clients that should be directed to it are redirected 
to one and a same other server without a need to sync states between 
servers.

This scheme has some disadvantages also - on certain circumstances, 
different sessions to one mailbox could be handled by different
servers in parallel. My tests showed that this causes some 
performance degradation but no index corruptions here (using OCFS2, 
not NFS). 

So my choice was to trade correctness (no parallel sessions to 
different servers) for simplicity (no state synchronization between 
servers).

> 
> Finally, there are the doveadm commands that can be used to:
> 
> 1) List the director status:
> # doveadm director status
> mail server ip	vhosts	users
> 11.22.3.44		100		1312
> 12.33.4.55		50		1424
> 
> 1) Add a new mail server (defaults are in dovecot.conf):
> # doveadm director add 1.2.3.4
> 
> 2) Change a mail server's vhost count to alter its connection count
(also
> works during adding):
> # doveadm director add 1.2.3.4 50
> 
> 3) Remove a mail server completely (because it's down):
> # doveadm director remove 1.2.3.4
> 
> If you want to slowly get users away from a specific server, you can
> assign its vhost count to 0 and wait for its user count to drop to zero.
If
> the server is still working while "doveadm director remove" is called,
new
> connections from the users in that server are going to other servers
while
> the old ones are still being handled.

This is nice admin interface.
Also, I have a question. Your implementation, what kind of sessions does 
it balance? I suppose imap/pop3. Is there a plan for similar redirecting
of LMTP connections based on delivery address?

Best regards and thanks for the great work
luben