[Dovecot] LSUB/SUBSCRIBE under namespaces

[bhayden at umn.edu]

Hi folks. 

Here's our situation: Migrating from UW-IMAP. Have lots (as in, tens of 
thousands) of clients set up using '~/mail' as the IMAP root, and using 
subscriptions.

Dovecot mail_location is using '~' for mbox storage. We've disabled full 
filesystem access in Dovecot in order to use ACLs for shared folders. Hence 
we have a hidden namespace for backwards compatibility with those clients 
using '~/mail' (dovecot -n output is at the end of this message so you can 
see exact settings). Here's the problem. If the client is not taking 
advantage of this hidden namespace--say it's configured to use plain old 
'mail' as the root directory--LSUB and SUBSCRIBE work as I'd expect. 
Subscribing to a folder called 'foo' puts this in the user's subscriptions 
file:

mail/foo

And this (from Thunderbird's log) is the LSUB exchange:

48203776[ed8ebe0]: 2e03600:server.umn.edu:A:SendData: 6 lsub "" "mail/*"
48203776[ed8ebe0]: ReadNextLine [stream=ed8ef88 nb=26 needmore=0]
48203776[ed8ebe0]: 2e03600:serverl.umn.edu:A:CreateNewLineFromSocket: * 
LSUB () "/" "mail/foo"
48203776[ed8ebe0]: ReadNextLine [stream=ed8ef88 nb=38 needmore=0]
   (trimmed...)
48203776[ed8ebe0]: 2e03600:bhayden.email.umn.edu:A:CreateNewLineFromSocket: 
6 OK Lsub completed.

All is well. However, when the client is configured with '~/mail' as the 
root and hence triggers the hidden namespace for such, SUBSCRIBE only 
writes a path relative to that namespace prefix. So as opposed the 
'mail/foo' in the first example, we get:

foo

written to the subs folder.

This leads to three problems:

1. Folders subscribed to when using this namespace will either disappear or 
break in a client using a different IMAP root setting that actually refers 
to the same place. For example, if this user moves to a client (such as 
both of the webmail offerings on campus) that use 'mail/' as the prefix, 
these folders will not be visible.

This is a fairly sizeable population of our users, whom we aren't 
necessarily in the position to say "too bad" to. :)

2. Users moved over from UW-IMAP, who were using '~/mail' for the IMAP 
root, have a subscriptions file full of:

~/mail/bar

Dovecot, with the same client configuration (since it's expecting a bare 
folder name to which it will prepend the namespace prefix), interprets this 
as so:

49036288[fa7e740]: 2eabc00:server.umn.edu:A:CreateNewLineFromSocket: * LSUB 
() "/" "~/mail/~/mail/foobar"

Leading to having the folder displayed in the client's list, but giving an 
error on SELECT. Now, this one we could theoretically "solve" by scrubbing 
the prefixes out of subs files as we convert; however, that leaves us with 
#1 above, where the bare folder name in the subs file depends on (to the 
average user) an obscure client config setting, which in practice in a 
large heterogeneous environment is unreliable.

3. Users who have been using an IMAP root of 'mail' in some client and 
subscribed to folders, who then use a client with the root set to '~/mail' 
see what appears to be "duplicates" of the same folder at the top level and 
then within a bogus 'mail/' subdirectory. They (quite reasonably, from 
their perspective), delete one of these, only to find they've deleted the 
only copy of said folder. Not good.

Also, users can't solve some of these problems by unsubbing and re-subbing 
on their own, as trying to unsubscribe from a folder that doesn't exist 
(such as ~/mail/~/mail/foobar in the example above) fails.

So, I'm wondering if there's any way to address this. I looked through the 
list archives and code pretty thoroughly. Only thing I noticed was a CVS 
log mentioning a "subscriptions=yes" setting for namespaces, which sounded 
like it might address this problem. However, putting that setting into 
dovecot.conf causes it to fail to start with a "Unrecognized config option" 
error, and using an imap_executable wrapper that sets the env variable as 
specified in the code (NAMESPACE_<number>_SUBSCRIPTIONS) sets it, as I can 
see from the dovecot log output:

Namespace: type=private, prefix=~/mail/, sep=/, inbox=no, hidden=yes, 
subscriptions=yes

But doesn't seem to have any effect. These two things combined lead me to 
believe this option was deprecated at some point?

Any thoughts or suggestions? While we're moving away from UW for a reason 
;), it does seem to handle this one thing more coherently as it recognizes, 
for example, that if the server equivalent of mail_location's storage is 
'~', then subscription file entries: '~/mail/foo' and 'mail/foo' are the 
same thing, while 'foo' is '~/foo'--regardless of client settings or 
namespaces. Whereas in the same situation Dovecot handles a client IMAP 
root of '~/mail' vs 'mail' differently, even though they are in fact the 
same thing.

This is also internally inconsistent, because (as noted above) when a 
namespace is not in effect Dovecot SUBSCRIBEs using a path relative to the 
mail_location storage--including the client's IMAP root portion of the 
path--whereas when a namespace is in effect it SUBSCRIBEs with only the 
bare mailbox name without the current client's IMAP root to give it context 
and meaning. Wouldn't it seem better to have these be consistent in this 
respect?

We need to figure this out somehow before going any further with our 
migration. We'll modify the source if necessary, but are hoping for a 
solution that doesn't require maintaining local mods to the source code 
unless we absolutely have to. :) And this seems like a basic enough issue 
that it'd be good to handle it in the real codebase. Thanks for any help or 
insight,

-Brian Hayden
Internet Services
University of Minnesota

dovecot -n output: (The shell script mail_executable doesn't come into play 
here--it sets some namespace settings only for special shared accounts 
using Maildir, and doesn't touch anything for the normal mbox accounts 
discussed above.)

# /etc/opt/dovecot/dovecot.conf syslog_facility: local6 protocols: imap 
imaps pop3 pop3s ssl_cert_file: /etc/opt/openssl/certs/dovecot.pem 
ssl_key_file: /etc/opt/openssl/private/dovecot.pem shutdown_clients: no 
login_dir: /var/opt/dovecot/run/dovecot/login login_executable(default): 
/opt/dovecot/libexec/dovecot/imap-login login_executable(imap): 
/opt/dovecot/libexec/dovecot/imap-login login_executable(pop3): 
/opt/dovecot/libexec/dovecot/pop3-login max_mail_processes: 8192 
first_valid_uid: 99 mail_location: 
mbox:~:INBOX=/var/mail/%u:INDEX=/var/mail/.dovecot-index/%1u/%u/ 
mail_debug: yes mail_executable(default): 
/opt/dovecot/libexec/dovecot/mail_path.sh mail_executable(imap): 
/opt/dovecot/libexec/dovecot/mail_path.sh mail_executable(pop3): 
/opt/dovecot/libexec/dovecot/pop3 mail_plugins(default): zlib 
mail_plugins(imap): zlib mail_plugins(pop3): mail_plugin_dir(default): 
/opt/dovecot/lib/dovecot/imap mail_plugin_dir(imap): 
/opt/dovecot/lib/dovecot/imap mail_plugin_dir(pop3): 
/opt/dovecot/lib/dovecot/pop3 imap_client_workarounds(default): 
delay-newmail outlook-idle tb-extra-mailbox-sep 
imap_client_workarounds(imap): delay-newmail outlook-idle 
tb-extra-mailbox-sep imap_client_workarounds(pop3): outlook-idle 
pop3_uidl_format(default): pop3_uidl_format(imap): pop3_uidl_format(pop3): 
%08Xu%08Xv namespace:
  type: private
  separator: /
  inbox: yes
namespace:
  type: private
  separator: /
  prefix: ~/mail/
  location: mbox:%h/mail/:INDEX=/var/mail/.dovecot-index/%1u/%u/
  hidden: yes
namespace:
  type: private
  separator: /
  prefix: ~%u/mail/
  location: mbox:~%u/mail/:INDEX=/var/mail/.dovecot-index/%1u/%u/
  hidden: yes
namespace:
  type: private
  separator: /
  prefix: Shared/
  location: 
maildir:~/Maildir/Shared:INDEX=/var/mail/.dovecot-index/%1u/u/shared auth 
default:
  username_format: %Lu
  passdb:
    driver: pam
    args: session=yes *
  userdb:
    driver: passwd
plugin:
  acl: vfile: