On 19.5.2010, at 16.16, luben karavelov wrote:

On Wed, 19 May 2010 10:51:06 +0200, Timo Sirainen tss@iki.fi wrote:

...

The company here in Italy didn't really like such idea, so I thought about making it more transparent and simpler to manage. The result is a new "director" service, which does basically the same thing, except without SQL database. The idea is that your load balancer can redirect connections to one or more Dovecot proxies, which internally then figure out where the user should go. So the proxies act kind of like a secondary load balancer layer.

As I understand, the first load balancer is just IP balancer, not POP3/IMAP balancer, isn't it?

Right.

I have implemented similar scheme here with imap/pop3 proxy (nginx) in front of dovecot servers. What i have found to work best (for my conditions) as hashing scheme is some sort of weighted constant hash.

I guess you meant consistent hash? Yeah, I thought about that too first but simpler hash seemed .. simpler. :)

In this way you do not need to synchronize a state between balancers and proxies. If you add or remove servers very few clients get reallocated - num active clients/num servers. .. This scheme has some disadvantages also - on certain circumstances, different sessions to one mailbox could be handled by different servers in parallel.

That's the main thing I wanted to prevent with the director service, so I don't think consistent hashing would have made implementing it easier. Although it might have helped make the caching work a bit better when servers were added/removed.

So my choice was to trade correctness (no parallel sessions to different servers) for simplicity (no state synchronization between servers).

I would have liked to avoid the state sync too, but I like more the idea of having it work 100% perfectly in all conditions. :)

Also, I have a question. Your implementation, what kind of sessions does it balance? I suppose imap/pop3. Is there a plan for similar redirecting of LMTP connections based on delivery address?

It's a pretty generic service. Currently imap and pop3 use it, but it would be pretty easy to make LMTP proxy support it too. The main difference is that for imap/pop3 it needs to emulate being an auth socket, while for lmtp it would need to emulate being a userdb socket. I'd guess it would need less than 50 lines of code total. I guess I should do it before v2.0.0.

On 25.5.2010, at 20.32, Brad Davidson wrote:

I have some questions about the suggested configuration, as well as the current implementation.

• Does this work for POP3 as well as IMAP?

Yes.

• Is there any reason not to use all 12 of our servers as proxies as well as mailbox servers, and let the director communication route connections to the appropriate endpoint?

So instead of having separate proxies and mail servers, have only hybrids everywhere? I guess it would almost work, except proxy_maybe isn't yet compatible with director. That's actually a bit annoying to implement.. You could of course run two separate Dovecot instances, but that also can be a bit annoying.

• Does putting a host into 'directed proxy' mode prevent it from servicing local mailbox requests?

No. The director service simply adds "host" field to auth lookup replies if the original reply had proxy=y but didn't have host field.

• How is initial synchronization handled? If a new host is added, is it sent a full copy of the user->host mapping database?

Yes. So the connections between the proxies should be pretty fast. I think the maximum bytes transferred per user is 38.

• What would you think about using multicast for the notifications instead of a ring structure? If we did set up all 12 hosts in a ring, it would be conceivable that a site failure plus failure of a single host at the surviving site would segment the ring. Multicast would prevent this, as well as (conceivably) simplifying dynamic resizing of the pool.

The proxies always try to keep connecting to next available server (i.e. if the next server won't connect, it tries one further away until it finally connects to something or reaches itself). So the segmentation could happen only if there was no network connection between the two segments.

I don't know much about how to do multicasting, except I've heard it can be problematic. Also not everything about the ring is simply about broadcasting user states. Most importantly the ring synchronization step during mail server adds/removes that prevents user from being assigned to different servers by different proxies wouldn't work with multicasting.

On ma, 2010-05-31 at 05:02 -0700, Brandon Davidson wrote:

...

So instead of having separate proxies and mail servers, have only hybrids everywhere? I guess it would almost work, except proxy_maybe isn't yet compatible with director. That's actually a bit annoying to implement.. You could of course run two separate Dovecot instances, but that also can be a bit annoying.

Would I have to run two separate instances, or could I just set up multiple login services on different ports; one set to proxy (forwarding the password to the remote server) and one set to not? I suppose each login service would have to use a different authdb, which I don't know how to do.

Well .. maybe you could use separate services. Have the proxy listen on public IP and the backend listen on localhost. Then you can do:

local_ip 127.0.0.1 { passdb { .. } }

and things like that. I think it would work, but I haven't actually tried.

...

No. The director service simply adds "host" field to auth lookup replies if the original reply had proxy=y but didn't have host field.

Interesting. It sounds like proxying requires a database query that will return 'proxy=y' as part of the auth lookup. It would be nice to have a static password authdb for proxying that didn't require a real database backend. I'm using PAM now, and don't see a good way to enable proxying.

The wiki also says that there's a way to let the proxy backend handle authentication, but I don't see an example of that anywhere.

There's not yet a static passdb .. perhaps there should be. But you could use e.g. sqlite backend for the proxy and use:

password_query = select null as password, 'Y' as nopassword, 'Y' as proxy

It looks like the mailserver list is initially populated from director_mail_servers, but can be changed by discovering hosts from other directors or by adding/removing hosts with doveadm. Since the initial host list is not written back in to the config file, changes made with doveadm are not persistent across service restarts.

Right. I considered using a more permanent database for them, but .. I don't know if that's a good idea.

Does 'doveadm director ' need to be run against each director individually, or will the changes be sent around the ring?

The changes are sent around the ring.

If a new host comes up with a mailserver in its list that has been removed by doveadm, will the handshake remove it from the list?

If a new director joins an existing ring, it ignores its own mail server list and uses the ring's.

The list of director servers used to build the ring is read from director_servers, and cannot be changed at runtime. A host finds its position within the ring based on its order within the list, and connects to hosts to its left and right until it has a connection on either side and can successfully send a test message around the ring.

The way directors can be added currently is:

1. Add the new host to one director's config.
2. Reload config, this should restart director process and have it reconnect to the ring, announcing the new host to everyone
3. Start up the new host, it'll insert itself to the ring.

That's anyway the theory, haven't tried yet. :) Also of course it's a good idea to add the host to all directors' config, but only one of them needs to be restarted to have it reload its config.

Is that all correct? What happens if some hosts have only a subset, or different subsets, of a group of hosts in their mail server or director server list?

That should never happen (well, except temporarily).

On 31.5.2010, at 22.31, stefan novak wrote:

is this proxy working for lmtp too?

Not yet, but it's probably less than 50 lines of code.

is there a roadmap when this will be released and considered as stable?

Hopefully by the time v2.0.0 is released. It should already be usable with two proxy servers.

On 31.5.2010, at 23.59, Brandon Davidson wrote:

You need to put the other passdb/userdb to the external IP:

local 1.2.3.4 {

userdb { driver = passwd } passdb { driver = sql args = /etc/dovecot/proxy-sqlite.conf }

}

Even if the alternate passdb worked, how would I get it to connect to the backend on localhost? It looks like the proxy connection comes in over the external IP even if it's to itself, as the external address is what's specified as the proxy destination by the director.

Yeah, you're right.. You could have it be listening on a different port. But there is no local_port {} block yet. Well, unless you changed the sqlite config so that it has ".. where %a=143". Then also return "14300 as port" or something.

I do have a private network that I run NFS over; I suppose I could run the proxy on the external, backend on the internal, and use only the internal IPs in the mailserver list. I've also tried that, but it doesn't seem to work either due to the passdb setting not being honored within local|remote blocks.

I guess that'd work too.

Even if it did, wouldn't it still complain about the proxy looping back to itself since both lip and rip would both be local addresses? Unless the loopback check just compares to see if they're the same...

That's how it does. lip=rip and lport=rport is required.

Either way, it seems like having proxy_maybe work with the director service would make the whole setup a lot simpler.

Hmm. I guess it could work like:

1. Director forwards auth lookup
2. Receives proxy=y with no host (auth process forgets about this request - user can't login with it)
3. Gets the remote IP
4. Figures out that it is the same IP and port where client connected
5. Either it has the username+password or master user+username+master password
6. It does another auth lookup, also giving some input parameter so that auth process will be forced to ignore the proxy and keep the request around so user can login
7. Director forwards the reply to login process, dropping the proxy stuff

The main thing to be implemented would be the "some input parameter". Maybe it could be just that it sets rip=lip and rport=lport and you could compare those (or just lip=rip) in the sql query.

On 1.6.2010, at 0.30, Brandon Davidson wrote:

May 31 16:20:42 cc-popmap7 dovecot: auth: Fatal: No passdbs specified in configuration file. PLAIN mechanism needs one

Hmm. Maybe this check should check also the local etc. blocks..

So I added a global passdb/userdb:

The passdbs and userdbs are checked in the order they're defined. You could add them at the bottom. Or probably more easily:

local 128.223.143.138 { passdb { driver = sql args = .. }

passdb { driver = pam } userdb { driver = passwd }

This should work because all proxy lookups get caught by the sql lookup, while non-proxy connections go directly to pam. And only non-proxy connections care about userdb.

On 1.6.2010, at 0.59, Brandon Davidson wrote:

Interestingly enough, if I run 'doveconf -n' it doesn't seem to be retaining the order I specified. The local section is dropped down to the very end:

Right .. it doesn't work exactly like that I guess. Or I don't remember :) Easiest to test with:

doveconf -f lip=128.223.142.138 -n

That should show the wanted order.

Timo,

On 5/31/10 5:34 PM, "Brandon Davidson" brandond@uoregon.edu wrote:

Still not sure why it's not proxying though. The config looks good but it's still using PAM even for the external IP.

I played with subnet masks instead of IPs and using remote instead of local, as well as setting auth_cache_size = 0, but no dice. It still seems to ignore the block and only use the global definition, even if doveconf -f lip=<ip> shows that it's expanding it properly.

-Brad

Timo,

On 5/31/10 6:56 PM, "Timo Sirainen" tss@iki.fi wrote:

Oh, you're right. For auth settings currently only protocol blocks work. It was a bit too much trouble to make local/remote blocks to work. :)

That's too bad! Any hope of getting support for this and director+proxy_maybe anytime soon?

-Brad

Timo,

-----Original Message-----

...

That's too bad! Any hope of getting support for this

I wasn't really planning on implementing it soon.

...

and director+proxy_maybe anytime soon?

I tried looking into it today, but it's an annoyingly difficult change, so probably won't happen soon either.

That's too bad! I guess I'll see what I can do within the current constraints. You'd gotten my hopes up though ;)

I was just playing around a bit, and came up with something like:

passdb { driver = sql args = /etc/dovecot/proxy-sqlite.conf } passdb { driver = pam }

driver = sqlite connect = /dev/null password_query = SELECT null AS password, 'Y' AS nopassword, 'Y' AS proxy WHERE '%{lip}' LIKE '10.142.0.%%' AND '%{lip}' != '%{rip}'

This is a hack, but should be roughly equivalent to proxy_maybe and a local block around the sql passdb, right?

-Brad

On Wed, 2010-06-16 at 15:19 +0200, Oliver Eales wrote:

Am 19.05.2010 10:51, schrieb Timo Sirainen:

...

As http://wiki.dovecot.org/NFS describes, the main problem with NFS has always been caching problems. One NFS client changes two files, but another NFS client sees only one of the changes, which Dovecot then assumes is caused by corruption.

host = vhosts[ md5(username) mod vhosts_count ]

Hello Timo, i am currently playing around with the new director service and i am really looking forward for it in 2.0 Wouldn't it be better to use a consistent hash function instead of the md5 ? So that you would only get a new assignment of users belonging to the failed server and not a "complete remapping".

I thought about it first, but then thought it would only make it more complex without much benefits.

With this setup it might be possible to store local indexes in a NFS Backend setting, as the users stay kind of sticky to their server.

Well, this would be a benefit that I didn't think of. :) Maybe it could be done in future.

And there would also be no need for the distribution of the currently active mappings within the ring. Maybe only for the state for the servers.

I think it's still needed. Maybe not all mappings, but at least when new servers are added. For example if server2 handles users A and B, and a new server3 is added. Now server2 would handle A and server3 would handle B. But server1 would still need to know that server2 would still have to handle B until it timeouts (B user can't just be moved to server3 without killing its existing connections, which doesn't sound like a good idea).