Maybe for v2.1:
imap and pop3 processes can currently handle multiple connections, but they're assigned pretty much randomly to processes. It would be better if it was possible to:
a) Require that a process handles only a specific user's connections
b) Require that a process handles only a specific UNIX UID's connections (e.g. if each domain has a separate UID)
c) Prefer to put user's connections to same process, but don't require it.
To do this, there needs to be a process between imap/pop3-login and imap/pop3. Let's call it imap/pop3-tracker process (or aggregator? or any suggestions?). This process accepts all connections coming from login processes when they've finished authenticating a connection. Internally it figures out if the connection should go to a new or an existing process. It keeps track of
username -> { mail_process[], cookie (see idle processes below) } mail_process -> { unix_socket_fd, connection_count }
So there's a connection to each mail process. This connection is used to send connection fds for the process to handle. It's also used to receive notifications when a connection is closed.
The mail process sockets could also be used to communicate to a specific process. For example doveadm could use this in future to ask what a specific user or process is currently doing, by connecting to imap-tracker, which forwards the request to wanted process(es).
The above isn't really anything new, just a more specific plan about what I had been thinking for a long time. A new and more interesting idea is to gather IDLEing imap connections under a few imap-idle processes:
When a connection has been IDLEing for a few minutes (or something), and there's no special state in the connection (e.g. CONTEXT=SEARCH searches), the connection could be moved to a separate imap-idle process that simply waits for something to happen. When something does happen, it'll move the connection back to imap process, which restores state and continues. The idea would be to reduce the number of idling imap processes that are wasting memory.
The state that needs to be saved and restored is:
- selected mailbox name, GUID, UIDVALIDITY (for reliably selecting back the original mailbox, or reliably fail if it's gone)
- flag: mailbox is EXAMINEd, not SELECTed
- highest seen modseq
- UIDs of \Recent messages
- SEARCHRES extension: last saved SEARCH result
- List of keywords that had been advertised in FLAGS/PERMANENTFLAGS reply to client. Or probably MD5 sum of it would be enough. If MD5 had changed, the list would be re-sent to client.
- connection fd
- paths that are being monitored with [id]notify/kqueue
Connection is moved back when a) new command is received from connection fd, b) one of the monitored files changes. The connections are moved back via imap-tracker process, by sending the saved state along the request. For security there's a per-user security cookie (128 bits of randomness), which is sent to imap processes, and when restoring connection from imap-idle, the request must provide the correct cookie or imap-tracker fails the login. (Normally new post-login imap/pop3 connections can't be created without having authenticated first via dovecot-auth. They verify it.)
So when connection comes back, imap process opens the mailbox based on saved information. If this fails for some reason (e.g. mailbox deleted), client gets disconnected. Next Dovecot gets a list of changes to mailbox since the saved modseq, and based on that sends the missing FETCH, EXPUNGE, EXISTS and RECENT events. Again if it somehow fails, the client gets disconnected.
At least initially this would only be done for connections that are running IDLE command, because non-IDLE connections are a lot more difficult to restore to their original state. This is mainly because if a message gets expunged, it's still visible to the connection until it issues a command that allows sending EXPUNGE event. To handle that, there would have to be code that allows building a view of mailbox where there are some expunged messages. Also it can be difficult to find out the expunged messages' flags, unless they're also part of the saved state.