Foreman component

[Timo Sirainen]

On 19 Apr 2016, at 12:55, Aki Tuomi <aki.tuomi at dovecot.fi> wrote:
> 
> I am planning to add foreman component to dovecot core and I am hoping
> for some feedback:
> 
> Foreman - generic per-user worker handling component

First an explanation of what this was planned to be used for: Think about many short-lived JMAP (HTTP) connections with each connection creating a new jmap process that opens the user's mailbox, processes the JMAP command, closes the mailbox and kills the process. Repeat for each command. Not very efficient when the same jmap process could handle all of the user's JMAP requests. The same problem exists also with most webmails' IMAP connections that are very short-lived.

One annoying problem with the foreman concept is that it requires an open UNIX socket for all the worker processes. Which could mean >10k open UNIX sockets, which all too often runs into file descriptor limits. We could of course just increase it high enough, and it probably would work ok.. But I also hate adding more of these "master" processes because they don't scale easily to multiple CPUs so they might become bottlenecks at some point (and some of these existing master processes already have become bottlenecks).

I've been trying to figure out a nice solution for the above problem for years already, but never really came up with anything better. Except today finally I had the new realization that anvil process already contains all of the needed information. We don't need a new process containing duplicated data, just some expansion of anvil and master. Of course, anvil is still kind of a "master" process that knows about all users, but it's already there anyway. And there's the new idea of how to avoid a single process using a ton of sockets:

(Talking only about IMAP here for clarity, but the same applies to POP3, JMAP and others.)

 - Today anvil already keeps track of (user, protocol, imap-process-pid), which is where "doveadm who" gets the user list.
 - Today imap-login process already does anvil lookup to see if the user has too many open connections. This lookup could be changed to also return the imap-process-pid[] array.
 - We'll add a new feature to Dovecot master: Ability to specify service imap { unix_listener /var/run/dovecot/login/imap-%{pid} { .. } }, which would cause such a UNIX socket path to be dynamically created for each created process. Only that one process is listening in the socket, master process itself wouldn't keep it open. When the process gets destroyed, the socket gets deleted automatically.
 - When imap process starts serving an IMAP connection, it does fchmod(socket, 0) for its imap-%{pid} listener. When it stops serving an active IMAP connection it does fchmod(socket, original-permissions).
 - imap-login process attempts to connect to each imap-%{pid} socket based on the imap-process-pid[] list returned by anvil. It ignores each EACCES failure, because those are already serving IMAP connections. If it succeeds in connecting, it sends the IMAP connection fd to it. If not, it connects to the default imap socket to create a new process.
 - The above method of trying to connect to every imap-process-pid[] is probaly efficient enough, although it probably ends up doing a lot of unnecessary connect() syscalls to sockets that are already handling existing connections. If this needs to be optimized, we could also enhance anvil to keep track of the "does this process have an active connection" flag and it would only return imap-process-pid[] for the processes without an active connection. There are of course some race conditions with this in any case but the worst that can happen is that a new imap process is created when there was another existing one already that could have served the connection, so slightly worse performance in some rare situations.

These same per-process sockets might be useful for other purposes too.. I've many times wanted an ability to communicate with an existing process. The "ipc" process was an attempt to do something about it, but it's not very nice and has the same problems with potentially using a huge number of fds.

Then there's the issue of how the management of idle processes (= processes with no active IMAP connections) goes:
 - service { idle_kill } already specifies when processes without clients are killed. We can use this here as well, so when IMAP connection has closed the process stays alive for idle_kill number of seconds until it gets closed.
 - If idle_kill times are set large enough on a busy system, we're usually reaching service { process_limit } constantly. So when no new processes can be created, we need the ability to kill an existing process instead. I think this is master process's job. When connection comes to "imap" and process_limit is reached, master picks the imap process with the longest idle-time and kills it (*). Then it waits for it to die and creates a new process afterwards. There's race condition here though and the process may not die but instead notify master that it's serving a new client. In this case master needs to retry with the next process. The process destroying might also not be fast always. To avoid unnecessarily large latencies due to waiting for process destruction, I think master should always try to stay a bit below process_limit (= a new service setting).
 - (*) I'm not sure if longest idle-time is the ideal algorithm. Some more heuristics would be useful, but this complicates master process too much. The processes themselves could try to influence master's decisions with some status notifications. For example if we've determined that user at example.com constantly logs in every 5 minutes, and the process has been idle for 4mins59 seconds, which is also the oldest idling process, we still don't want to kill it because we know that it's going to be recreated in 1 second anyway. This is probably not going to be in the first version though.