Home directories with colons in it

Thu Jun 16 10:39:33 UTC 2022

Howdy folks,

we're using Dovecot 2.3.19 in combination with vmailmgr as the virtual user management backend (please don't judge, there are plans to move away from it for various reasons, but as of today it's there), using the wrapper from https://wiki.dovecot.org/HowTo/VMailMgr - and we're facing an interesting problem with colons in home directories. I think this problem is not specific to vmailmgr but happens in any scenario that includes ":" in directory names.

tl;dr:

- The checkvpw-dovecot-wrapper from the Dovecot wiki converts ":" in directory names to "::" which works fine with general IMAP use, but at least the autoexpunge plugin and ManageSieve don't handle this escaping and thus break on those users.

- When patching the ":"->"::" conversion out of the checkvpw-dovecot-wrapper, Dovecot itself bails on it, interpreting it as a sub-setting.

- The latter is quite surprising because a HOME/mail_home directory is supposed to be exactly one directory (AFAIK), with no sub-settings like e.g. mail_location, so I did not expect it to be "interpreted" in any way.

Short story long:

When you set up a user with a name containing dots, vmailmgr converts them to colons, pretty much like qmail does (qmail maps the username "xy.z" to a file named ".qmail-xy:z"; vmailmgr creates the directory "./users/xy:z" for that username). The wrapper from the dovecot wiki escapes ":" into "::" before exporting the user's directory as the HOME environment variable:

        /* escape possible ':' characters before finally exporting */
        escape_colon(buffer, MAXLEN);

        setenv("HOME", buffer, 1);

The result looks like that:

[root at stardust ~]# doveadm auth login xy.z at coltest.uber.space supersecret
passdb: xy.z at coltest.uber.space auth succeeded
extra fields:
  user=xy.z at coltest.uber.space

userdb extra fields:
  xy.z at coltest.uber.space
  home=/home/coltest/./users/xy::z   <---------- notice the double colon here
  uid=1009
  gid=1010
  auth_mech=PLAIN

So far, so good: Working with IMAP generally works fine, but there's a catch: Not every module seems prepared for this "::" syntax. The first notable exception is the autoexpunge plugin which simply bails on it:

Error: autoexpunge: Couldn't create dovecot.autoexpunge.lock lock: file_create_locked(/home/coltest/./users/xy::z/dovecot.autoexpunge.lock) failed: safe_mkstemp(/home/coltest/./users/xy::z/dovecot.autoexpunge.lock) failed: No such file or directory

The second notable exception is ManageSieve which takes the escaped directory name "as is", leading to a situation where the Maildir is in "xy:z" while the Sieve stuff is being put in "xy::z" where it effectively is never picked up during delivery (as the stuff is in the wrong location and the .dovecot.sieve symlink is - silently - not created):

[coltest at stardust ~]$ ls -la users/*
users/xy:z:
total 28
drwx------. 9 coltest coltest 4096 Jun 16 11:39 .
drwxr-xr-x. 4 coltest coltest   31 Jun 16 11:40 ..
drwxr-xr-x. 2 coltest coltest    6 Jun 16 11:39 cur
-rw-------. 1 coltest coltest  376 Jun 16 11:40 dovecot.index.log
-rw-------. 1 coltest coltest 2676 Jun 16 11:39 dovecot.list.index.log
-rw-------. 1 coltest coltest   96 Jun 16 11:39 dovecot.mailbox.log
-rw-------. 1 coltest coltest   51 Jun 16 11:39 dovecot-uidlist
-rw-------. 1 coltest coltest    8 Jun 16 11:39 dovecot-uidvalidity
-r--r--r--. 1 coltest coltest    0 Jun 16 11:39 dovecot-uidvalidity.62aafa70
drwx------. 5 coltest coltest  108 Jun 16 11:39 .Drafts
drwxr-xr-x. 2 coltest coltest    6 Jun 16 11:39 new
drwx------. 5 coltest coltest  108 Jun 16 11:39 .Sent
drwx------. 5 coltest coltest  108 Jun 16 11:39 .Spam
-rw-------. 1 coltest coltest   28 Jun 16 11:39 subscriptions
drwxr-xr-x. 2 coltest coltest    6 Jun 16 11:39 tmp
drwx------. 5 coltest coltest  108 Jun 16 11:39 .Trash

users/xy::z:
total 0
drwx------. 3 coltest coltest 19 Jun 16 11:40 .
drwxr-xr-x. 4 coltest coltest 31 Jun 16 11:40 ..
drwx------. 3 coltest coltest 33 Jun 16 11:40 sieve

So my initial thought was: Why is the ":" even escaped to a "::" in the first place? It shouldn't be necessary according to this clean workaround that I found in a posting from 2009 with Timo advising for this type of handling:

> Hmm. Although now that I looked at your code, you seem to be using only
> HOME. And using ~/ in mail_location should work around any ":" issues.
> So what exactly is the problem you're seeing?

https://www.dovecot.org/list/dovecot/2009-June/040590.html

(In the same thread, Timo _also_ introduced the "::" handling in Dovecot.)

This is exactly what we're doing at delivery time: Setting mail_home to the real pathname including a _single_ colon (not an escaped double colon) and then pointing mail_location to "~/":

/usr/libexec/dovecot/dovecot-lda -a '$RECIPIENT' -f '$SENDER' -o mail_home='$MAILDIR' -o mail_location='maildir:~/'

This works well.

So my assumption was: If the checkvpw-dovecot-wrapper returns the HOME directory of a virtual user; where's even the need to escape a single colon to a double colon? So I gave it a try and removed that one line of escaping and recompiled the wrapper. Result:

[root at stardust ~]# doveadm auth login xy.z at coltest.uber.space supersecret
passdb: xy.z at coltest.uber.space auth succeeded
extra fields:
  user=xy.z at coltest.uber.space

userdb extra fields:
  xy.z at coltest.uber.space
  home=/home/coltest/./users/xy:z   <---------- single colon here now
  uid=1009
  gid=1010
  auth_mech=PLAIN

Using a real IMAP login, though, this lead to the "interesting" situation that the login itself worked correctly, but _then_ immediately an "Internal error occurred":

[root at stardust ~]# nc localhost 143
* OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SPECIAL-USE LITERAL+ STARTTLS AUTH=PLAIN] Dovecot ready.
0 LOGIN xy.z at coltest.uber.space supersecret
0 OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS BINARY MOVE SNIPPET=FUZZY PREVIEW=FUZZY PREVIEW STATUS=SIZE SAVEDATE SPECIAL-USE LITERAL+ NOTIFY SPECIAL-USE] Logged in
* BYE Internal error occurred. Refer to server log for more information.

This is the according log:

Jun 16 10:04:37 stardust.uberspace.de dovecot[15268]: Jun 16 10:04:37 imap-login: Info: Login: user=<xy.z at coltest.uber.space>, method=PLAIN, rip=::1, lip=::1, mpid=22889, secured
Jun 16 10:04:37 stardust.uberspace.de dovecot[15268]: Jun 16 10:04:37 imap(xy.z at coltest.uber.space)<22889><utbow43hgJoAAAAAAAAAAAAAAAAAAAAB>: Error: Namespace '': Unknown setting: z
Jun 16 10:04:37 stardust.uberspace.de dovecot[15268]: Jun 16 10:04:37 imap(xy.z at coltest.uber.space)<22889><utbow43hgJoAAAAAAAAAAAAAAAAAAAAB>: Info: Disconnected: Namespace '': Unknown setting: z in=0 out=428 deleted=0 expunged=0 trashed=0 hdr_count=0 hdr_bytes=0 body_count=0 body_bytes=0

So it looks like Dovecot tries to do some interpretation on the HOME value, splitting it at the ":" and interpret the "z" as some kind of sub-setting - which a HOME variable (in contrast to, e.g. mail_location) is not supposed to have.

Here I'm a bit stuck. My idea is to work around this by eliminating the ":" from directory names altogether by passing an explicit directory name to vadduser:

[coltest at stardust ~]$ echo supersecret | vadduser --directory=./users/ab.c ab.c
vadduser: user 'ab.c' successfully added
[coltest at stardust ~]$ ls -l users
total 0
drwx------. 5 coltest coltest 39 Jun 16 10:26 ab.c
drwx------. 5 coltest coltest 39 Jun 16 09:31 xy:z

But still, it would be nicer if Dovecot cleanly supported the ":" in directory names, as it seems to _want_ to support it (by introducing the "::" escaping) but then only does it halfway.

Could this be fixed in whatever direction, or is there any other advice how we should handle this?

All the best,
Jonas