[Dovecot] Apple patch 10

[Mike Abbott]

Patch #10 allows the pop3 and imap mail processes to handle multiple  
clients.  We know this weakens the security model but it greatly  
increases scalability especially when clients are idle.

Here's how it works.  When there are no mail processes, or none  
serving fewer than mail_max_connections clients, the master creates a  
new mail process pretty much like it does now, but with some new  
environment variables (PERSISTENT_MAIL_PROCESS and CONNECTION_ID, and  
ADVISE_SET[GU]ID instead of RESTRICT_SET[GU]ID) and a unix-domain  
socket connection back to the master.  For future clients the master  
sends the client socket and a full dump of the environment via that  
connection.  Persistent mail processes multiplex I/O for their  
connected clients and switch the whole environment and the effective  
user ID, group ID, and supplemental groups each time.  The result is  
that most of dovecot's code and assumptions remain unchanged:  the  
mail process executes imap/pop commands with the permissions of the  
user and with nearly the same environment so getenv() still works as  
expected.  Some assumptions do change, for instance global state  
variables like last_partial and quota_set are forbidden, and must be  
stored per-client or per-user.  Also it's no longer OK just to exit or  
panic on error, for that disconnects multiple users not just one.

Some key entry points in the patch are create_mail_process() in mail- 
process.c and io_env_switch() in ioloop.c.

Notes about this patch:

- The base for this patch is dovecot-1.1.7 + Apple patch 9, not  
because this patch needs Open Directory but simply because it adds one  
line to the Apple attributions in COPYING.

- This patch depends on Apple patch 7 (hash_table_create/destroy).

- This patch introduces the following new config options for the pop3  
and imap protocols:
	- mail_process_per_connection = yes
	- mail_max_connections = 20
The default value of mail_process_per_connection (yes) preserves the  
current secure model.  Changing it to no allows multiplexing, with up  
to mail_max_connections simultaneous clients per process.  See dovecot- 
example.conf for more info.

- It restricts the mail_executable config option to minimize  
unpleasant surprises.  When mail_process_per_connection = no the  
communication protocol between the master process and the mail  
processes changes as described above.  A naive third-party  
mail_executable might break.  See dovecot-example.conf for more info.

- Some changes are intentionally awkward in order to minimize code  
deltas to simplify merges (of new dovecot releases into our source  
tree).  These are so marked.  Feel free to clean them up.

- Changes tagged with "APPLE" are ours, including the whole new  
directory mail-common.  All the untagged changes, including the entire  
contents of the new files mail-user.[ch], are copied straight from  
dovecot-1.2 (alpha3, I believe).  We needed dovecot-1.2's support for  
multiple users in order to handle quotas correctly, and we needed it  
in 1.1.  Specifically, we copied these changes from dovecot-1.2 (and a  
couple others too I think):
     8082 http://hg.dovecot.org/dovecot-1.2/rev/db66611fd195
     Added struct mail_user and fixed the code to support multiple  
users per process.
     8084 http://hg.dovecot.org/dovecot-1.2/rev/f12f8c1da0bf
     Forgot to add mail-user.* files in previous struct mail_user  
commit.
     8085 http://hg.dovecot.org/dovecot-1.2/rev/bf83aa9c3f4a
     Removed pool parameter from mail_namespaces_init*(). Use  
mail_user's pool intead.
     8091 http://hg.dovecot.org/dovecot-1.2/rev/ceca59aaae89
     quota-fs: compile fix for previous changes.
     8096 http://hg.dovecot.org/dovecot-1.2/rev/f35a8a3dc06d
     Fixed FS quota compiling and Maildir++ quota with multiple users.
     8109 http://hg.dovecot.org/dovecot-1.2/rev/e7929190cd32
     fts-solr: Fixed compiling with recent struct mail_user changes.
     8137 http://hg.dovecot.org/dovecot-1.2/rev/b2a258213ee0
     Created mail_user_[try_]home_expand(). Used them for expanding  
mail directories.
     8294 http://hg.dovecot.org/dovecot-1.2/rev/8aa69e3d27ef
     Trash plugin: Assign storage to all mailboxes at startup so  
errors are caught immediately.
     Also previous optimization change broke trash plugin when using  
multiple
     mail_users. This change fixes it to work again.
If a change in the patch is not marked somehow with APPLE then it  
comes from dovecot-1.2.

- Some other data structures also needed to be made per-client/user,  
for instance last_partial from imap-fetch-body.c, quota_set from quota- 
plugin.c, and process_primary_gid etc. from restrict-access.c.

- Sending the master process SIGINFO (on platforms that support that  
signal) makes it print its view of all the clients connected to all  
the mail processes.  Sending SIGINFO to an individual mail process  
makes it print its view.  These are not necessarily always the same.   
For instance, on a config reload (SIGHUP), the master disconnects from  
the mail processes but leaves them running.  The output is admittedly  
kinda geeky.

- The logging tag for a persistent mail process is just "*" instead of  
the user name, since a process can serve multiple users. Some  
individual messages (such as connect/disconnect) identify the user.

- The expire plugin uses global state variables which are incompatible  
with persistent mail processes.  The patch detects trouble but does  
not fix it because we don't need it.  We leave that as an exercise for  
you :).

- The dict client interface is blocking which is unfortunate for  
persistent mail processes.  The patch does not address this, since the  
delay is probably small and we don't use it anyway.

- Some of the accounting is gnarly because of the need to honor  
mail_max_userip_connections in a world where a single user can have  
any number of active connections on any number of mail processes.

Finally, here's the patch.

-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: patch10.txt
Url: http://dovecot.org/pipermail/dovecot/attachments/20090106/82abaed9/attachment-0001.txt 
-------------- next part --------------