[Dovecot] Allowing tilde at start of mailbox names
In lib-storage/index/maildir/maildir-storage.c maildir_is_valid_create_name() and maildir_is_valid_existing_name() the following sequence of tests appear but I don't really understand why if ((storage->flags & MAIL_STORAGE_FLAG_FULL_FS_ACCESS) != 0) return TRUE; if (*name == '~' || strchr(name, '/') != NULL) return FALSE; If MAIL_STORAGE_FLAG_FULL_FS_ACCESS is set then the validator funcs return TRUE immediately and code elsewhere will do appropriate home_exapand for mailbox names. However if MAIL_STORAGE_FLAG_FULL_FS_ACCESS is not set the (*name == '~') test prevents mailbox names that begin with '~'. Why is it not possible to have mailbox names that begin with '~' but without '~' carrying any special meaning? I have been working on a patch (appended but incomplete; the option only works against maildir atm) that adds a new option mail_allow_tilde and my admittedly limited testing seems to indicate that it does the right thing but maybe I am barking up the wrong tree. Am I missing something here? ---------------------------------------- # HG changeset patch # User <pod@sysdev.oucs.ox.ac.uk> # Date 1185368923 -3600 # Node ID ab9cf3790ea9b5486afa6498f60c6bf6fb2e5f1e # Parent a23be695672737967a10613d7dca2a6f53b5d96d create new option mail_allow_tilde diff -r a23be6956727 -r ab9cf3790ea9 src/lib-storage/index/maildir/maildir-storage.c --- a/src/lib-storage/index/maildir/maildir-storage.c Tue Jul 24 05:48:03 2007 +0300 +++ b/src/lib-storage/index/maildir/maildir-storage.c Wed Jul 25 14:08:43 2007 +0100 @@ -240,7 +240,9 @@ static bool maildir_is_valid_create_name if ((storage->flags & MAIL_STORAGE_FLAG_FULL_FS_ACCESS) != 0) return TRUE; - if (*name == '~' || strchr(name, '/') != NULL) + if ((*name == '~' && + (storage->flags & MAIL_STORAGE_FLAG_ALLOW_TILDE) == 0) || + strchr(name, '/') != NULL) return FALSE; if (name[0] == '.' && (name[1] == '\0' || @@ -264,7 +266,9 @@ static bool maildir_is_valid_existing_na if ((storage->flags & MAIL_STORAGE_FLAG_FULL_FS_ACCESS) != 0) return TRUE; - if (*name == '~' || strchr(name, '/') != NULL) + if ((*name == '~' && + (storage->flags & MAIL_STORAGE_FLAG_ALLOW_TILDE) == 0) || + strchr(name, '/') != NULL) return FALSE; if (name[0] == '.' && (name[1] == '\0' || diff -r a23be6956727 -r ab9cf3790ea9 src/lib-storage/mail-storage.c --- a/src/lib-storage/mail-storage.c Tue Jul 24 05:48:03 2007 +0300 +++ b/src/lib-storage/mail-storage.c Wed Jul 25 14:08:43 2007 +0100 @@ -66,6 +66,8 @@ void mail_storage_parse_env(enum mail_st *flags_r = 0; if (getenv("FULL_FILESYSTEM_ACCESS") != NULL) *flags_r |= MAIL_STORAGE_FLAG_FULL_FS_ACCESS; + if (getenv("ALLOW_TILDE") != NULL) + *flags_r |= MAIL_STORAGE_FLAG_ALLOW_TILDE; if (getenv("DEBUG") != NULL) *flags_r |= MAIL_STORAGE_FLAG_DEBUG; if (getenv("MMAP_DISABLE") != NULL) diff -r a23be6956727 -r ab9cf3790ea9 src/lib-storage/mail-storage.h --- a/src/lib-storage/mail-storage.h Tue Jul 24 05:48:03 2007 +0300 +++ b/src/lib-storage/mail-storage.h Wed Jul 25 14:08:43 2007 +0100 @@ -34,7 +34,9 @@ enum mail_storage_flags { fail to create the storage. */ MAIL_STORAGE_FLAG_NO_AUTOCREATE = 0x200, /* Rely on O_EXCL when creating dotlocks */ - MAIL_STORAGE_FLAG_DOTLOCK_USE_EXCL = 0x400 + MAIL_STORAGE_FLAG_DOTLOCK_USE_EXCL = 0x400, + /* Allow folder names to begin with '~' TILDE */ + MAIL_STORAGE_FLAG_ALLOW_TILDE = 0x800 }; enum mail_storage_lock_method { diff -r a23be6956727 -r ab9cf3790ea9 src/master/mail-process.c --- a/src/master/mail-process.c Tue Jul 24 05:48:03 2007 +0300 +++ b/src/master/mail-process.c Wed Jul 25 14:08:43 2007 +0100 @@ -254,6 +254,8 @@ mail_process_set_environment(struct sett env_put("DEBUG=1"); if (set->mail_full_filesystem_access) env_put("FULL_FILESYSTEM_ACCESS=1"); + if (set->mail_allow_tilde) + env_put("ALLOW_TILDE=1"); if (set->pop3_no_flag_updates) env_put("POP3_NO_FLAG_UPDATES=1"); if (set->pop3_reuse_xuidl) diff -r a23be6956727 -r ab9cf3790ea9 src/master/master-settings-defs.c --- a/src/master/master-settings-defs.c Tue Jul 24 05:48:03 2007 +0300 +++ b/src/master/master-settings-defs.c Wed Jul 25 14:08:43 2007 +0100 @@ -67,6 +67,7 @@ static struct setting_def setting_defs[] DEF(SET_INT, mailbox_idle_check_interval), DEF(SET_BOOL, mail_debug), DEF(SET_BOOL, mail_full_filesystem_access), + DEF(SET_BOOL, mail_allow_tilde), DEF(SET_INT, mail_max_keyword_length), DEF(SET_BOOL, mail_save_crlf), DEF(SET_BOOL, mail_read_mmaped), diff -r a23be6956727 -r ab9cf3790ea9 src/master/master-settings.c --- a/src/master/master-settings.c Tue Jul 24 05:48:03 2007 +0300 +++ b/src/master/master-settings.c Wed Jul 25 14:08:43 2007 +0100 @@ -216,6 +216,7 @@ struct settings default_settings = { MEMBER(mailbox_idle_check_interval) 30, MEMBER(mail_debug) FALSE, MEMBER(mail_full_filesystem_access) FALSE, + MEMBER(mail_allow_tilde) FALSE, MEMBER(mail_max_keyword_length) 50, MEMBER(mail_save_crlf) FALSE, MEMBER(mail_read_mmaped) FALSE, diff -r a23be6956727 -r ab9cf3790ea9 src/master/master-settings.h --- a/src/master/master-settings.h Tue Jul 24 05:48:03 2007 +0300 +++ b/src/master/master-settings.h Wed Jul 25 14:08:43 2007 +0100 @@ -75,6 +75,7 @@ struct settings { unsigned int mailbox_idle_check_interval; bool mail_debug; bool mail_full_filesystem_access; + bool mail_allow_tilde; unsigned int mail_max_keyword_length; bool mail_save_crlf; bool mail_read_mmaped;
On 25.7.2007, at 16.35, pod wrote:
In lib-storage/index/maildir/maildir-storage.c maildir_is_valid_create_name() and maildir_is_valid_existing_name()
the following sequence of tests appear but I don't really understand why
It's mostly there just to make sure that ~/ or ~user/ can't be used
to open mailboxes where you weren't supposed to have access to. I
didn't think anyone would really want to have mailboxes beginning
with ~. Do you really want them? (I don't think they were possible
with UW-IMAP either?)
Why is it not possible to have mailbox names that begin with '~' but without '~' carrying any special meaning?
That's what the code does if you don't have
mail_full_filesystem_access=yes, but I thought I'd prevent this in
=no case just in case of a bug somewhere, possibly even outside of
Dovecot itself.
I have been working on a patch (appended but incomplete; the option
only works against maildir atm) that adds a new option mail_allow_tilde
and my admittedly limited testing seems to indicate that it does the right
thing but maybe I am barking up the wrong tree. Am I missing something
here?
There are too many settings already. mail_full_filesystem_access=no
should be enough for this.
"TS" == Timo Sirainen <tss@iki.fi> writes:
TS> It's mostly there just to make sure that ~/ or ~user/ can't be
TS> used to open mailboxes where you weren't supposed to have access
TS> to. I didn't think anyone would really want to have mailboxes
TS> beginning with ~. Do you really want them? (I don't think they
TS> were possible with UW-IMAP either?)Unfortunately yes it would be useful in my case to have them because of historical accident. UW-IMAP does in fact allow it but maybe only if one is using particular build-time options such as blackBoxDir; I forget the precise details. Suffice to say at some point users were erroneously told to configure clients with '~/mail' as a prefix and this in combination with the particular UW-IMAP build options means we have users with folder hierarchies beginning "~/mail" ('~' uninterpreted). I don't think I can work around this with a hidden namespace since a client subsequently configured not to use a prefix will be used to seeing folders under "~/mail" ('~' uninterpreted). I'm (increasingly forlornly it seems) attempting to migrate with no user visible change.
>> Why is it not possible to have mailbox names that begin with '~'
>> but without '~' carrying any special meaning?
TS> That's what the code does if you don't have
TS> mail_full_filesystem_access=yes, but I thought I'd prevent this in
TS> no case just in case of a bug somewhere, possibly even outside of
TS> Dovecot itself.I appreciate your reasoning regarding bugs elsewhere. I was, however, hoping to introduce a mode of behaviour that would allow me to accept the risk. I'm not using virtual users; every login runs under its own uid so I still have filesystem permissions as a reasonable fall back.
TS> There are too many settings
TS> already. mail_full_filesystem_access=no should be enough for this.I completely understand why you want to keep the number of setting down. My motivation for introducing a new setting was a vain hope that allowing uninterpreted (*name == '~') behaviour could go into the 1.0 codebase but without also changing the meaning of mail_full_filesystem_access=no in existing deployments.
On 25.7.2007, at 19.13, pod wrote:
TS> It's mostly there just to make sure that ~/ or ~user/ can't be TS> used to open mailboxes where you weren't supposed to haveaccess TS> to. I didn't think anyone would really want to have mailboxes TS> beginning with ~. Do you really want them? (I don't think they TS> were possible with UW-IMAP either?)
Unfortunately yes it would be useful in my case to have them
because of historical accident.
How about if listescape plugin escaped it? That would make it
possible to use them but still safe against accidental problems.
For others too, I implemented a plugin for escaping mailbox names
with Maildir so that it's possible to use '.' in mailbox names:
http://dovecot.org/patches/1.0/listescape-plugin.c
"TS" == Timo Sirainen <tss@iki.fi> writes:
TS> How about if listescape plugin escaped it? That would make it
TS> possible to use them but still safe against accidental problems.Yes, I think that would work fine at least for my particular purposes since we'll almost certainly be running with the listescape plugin anyway.
TS> For others too, I implemented a plugin for escaping mailbox names
TS> with Maildir so that it's possible to use '.' in mailbox names:
TS> http://dovecot.org/patches/1.0/listescape-plugin.cNot one to blow your own trumpet are you! For further explanation for the benefit of the list: I recently contacted Timo privately asking if he would consider us funding him to implement a modified maildir layout for us. A few hours later he responded with listescape-plugin.c. What an absolute star.
On Wed, 2007-07-25 at 18:41 +0100, pod wrote:
"TS" == Timo Sirainen <tss@iki.fi> writes:
TS> How about if listescape plugin escaped it? That would make it TS> possible to use them but still safe against accidental problems.Yes, I think that would work fine at least for my particular purposes since we'll almost certainly be running with the listescape plugin anyway.
Updated the plugin.
On Wed, Jul 25, 2007 at 08:28:52PM +0300, Timo Sirainen wrote:
On 25.7.2007, at 19.13, pod wrote:
TS> It's mostly there just to make sure that ~/ or ~user/ can't be TS> used to open mailboxes where you weren't supposed to have access TS> to. I didn't think anyone would really want to have mailboxes TS> beginning with ~. Do you really want them? (I don't think they TS> were possible with UW-IMAP either?)Unfortunately yes it would be useful in my case to have them because of historical accident.
How about if listescape plugin escaped it? That would make it possible to use them but still safe against accidental problems.
For others too, I implemented a plugin for escaping mailbox names with Maildir so that it's possible to use '.' in mailbox names: http://dovecot.org/patches/1.0/listescape-plugin.c
I am using the 1.1 version of this plugin. So far it seems to be working fine. Is there a chance of having it be an official plugin distributed with 1.1? That would make it fit into my build environment alot cleaner, as well as a FreeBSD port I intend to submit so others can build 1.1 easily. One way or another I plan to keep using listescape and will need to find the best way to keep it functioning alongside dovecot upgrades, even if that means compiling and installing it manually each time. Thanks.
On Sun, Dec 02, 2007 at 03:01:51AM -0500, Adam McDougall wrote:
On Wed, Jul 25, 2007 at 08:28:52PM +0300, Timo Sirainen wrote:
On 25.7.2007, at 19.13, pod wrote:>> TS> It's mostly there just to make sure that ~/ or ~user/ can't be
TS> used to open mailboxes where you weren't supposed to have access TS> to. I didn't think anyone would really want to have mailboxes TS> beginning with ~. Do you really want them? (I don't think they TS> were possible with UW-IMAP either?)Unfortunately yes it would be useful in my case to have them because of historical accident.
How about if listescape plugin escaped it? That would make it possible to
use them but still safe against accidental problems.
For others too, I implemented a plugin for escaping mailbox names with
Maildir so that it's possible to use '.' in mailbox names:
http://dovecot.org/patches/1.0/listescape-plugin.cI am using the 1.1 version of this plugin. So far it seems to be working fine. Is there a chance of having it be an official plugin distributed with 1.1? That would make it fit into my build environment alot cleaner, as well as a FreeBSD port I intend to submit so others can build 1.1 easily. One way or another I plan to keep using listescape and will need to find the best way to keep it functioning alongside dovecot upgrades, even if that means compiling and installing it manually each time. Thanks.
Actually two issues came up. I think they are both involving folder listing.
In squirrelmail, if I am subscribed to #shared/decs with the listescape plugin enabled, the shared folders such as #shared/decs/network are greyed out. #shared/decs is a public namespace and the only reason I would be subscribed to it is for Outlook Express's sake or so Squirrelmail will show the shared folder hierarchy (same reason for subscribing to #shared, but being subscribed to #shared doesn't seem to hurt anything). I think squirrelmail sees the subscription to the folder (I can subscribe to unsubscribed shared folders too) but when it tries to LIST the folder, it gets nothing.
In all programs, it seems like listing folders in my additional private hidden namespaces comes up with nothing. a LIST "" "mail/%" a OK List completed. If I try to access a folder under mail/ directly by typing the path into mutt, it works.
with listescape:
a list "" "dot.test"
- LIST (\HasNoChildren) "." "dot.test" a OK List completed.
a LIST "" "mail/"
- LIST (\Noselect \HasChildren) "/" "mail/" a OK List completed.
a LIST "" "mail/%" a OK List completed.
(a LSUB "" "*" shows all the folders I am subscribed to, including #shared/decs/network)
a LSUB "" "#shared/%"
- LSUB () "/" "#shared/decs" a OK Lsub completed.
a LIST "" "#shared/decs"
- LIST (\Noselect \HasChildren) "/" "#shared/decs" a OK List completed.
a LSUB "" "#shared/decs/%" a OK Lsub completed.
a LIST "" "#shared/decs/network" a OK List completed.
(for some reason in squirrelmail, if I am unsubscribed from #shared/decs with the listescape plugin enabled, the shared folders such as #shared/decs/network are not greyed out and work, but I cant seem to easily reproduce that result with imap commands.
Without listescape loaded:
a list "" "dot\\2etest"
- LIST (\HasNoChildren) "." {10} dot\2etest a OK List completed.
a LIST "" "mail/"
- LIST (\Noselect \HasChildren) "/" "mail/" a OK List completed.
a LIST "" "mail/%"
- LIST (\HasNoChildren) "/" "mail/20041223"
- LIST (\HasNoChildren) "/" "mail/20050324"
- LIST (\HasNoChildren) "/" "mail/APA-MEA-DISCUSS"
- LIST (\HasNoChildren) "/" "mail/Drafts"
- LIST (\HasNoChildren) "/" "mail/IMAP"
- LIST (\HasNoChildren) "/" "mail/Junk E-mail" (etc, all the folders I expect)
a LSUB "" "#shared/%"
- LSUB () "/" "#shared/decs" a OK Lsub completed.
a LIST "" "#shared/decs"
- LIST (\Noselect \HasChildren) "/" "#shared/decs" a OK List completed.
a LSUB "" "#shared/decs/%"
- LSUB () "/" "#shared/decs/unixadmin"
- LSUB () "/" "#shared/decs/support"
- LSUB () "/" "#shared/decs/security"
- LSUB () "/" "#shared/decs/printmaster"
- LSUB () "/" "#shared/decs/postmaster"
- LSUB () "/" "#shared/decs/pcadmin"
- LSUB () "/" "#shared/decs/network"
- LSUB () "/" "#shared/decs/linuxadmin" a OK Lsub completed.
a LIST "" "#shared/decs/network"
- LIST (\HasChildren) "/" "#shared/decs/network" a OK List completed.
1.1.beta9: /usr/local/etc/dovecot.conf
ssl_cert_file: /usr/local/etc/apache2/ssl/mail.egr.msu.edu.pem ssl_key_file: /usr/local/etc/apache2/ssl/mail.egr.msu.edu.pem login_dir: /var/run/dovecot/login login_executable: /usr/local/libexec/dovecot/imap-login mail_max_userip_connections: 25 verbose_proctitle: yes first_valid_uid: 1000 first_valid_gid: 1000 mail_extra_groups: postlocal mail_location: maildir:%h/Maildir:CONTROL=%h/Maildir/dovecot/private/control:INDEX=%h/Maildir/dovecot/private/indexes mmap_disable: yes dotlock_use_excl: yes mail_nfs_storage: yes mail_nfs_index: yes mail_plugins: acl fts fts_squat listescape mail_log_max_lines_per_sec: 0 imap_client_workarounds: delay-newmail netscape-eoh tb-extra-mailbox-sep namespace: type: private separator: / inbox: yes list: yes subscriptions: yes namespace: type: private separator: / prefix: mail/ hidden: yes subscriptions: yes namespace: type: private separator: / prefix: Mail/ hidden: yes subscriptions: yes namespace: type: public separator: / prefix: #shared/decs/ location: maildir:/egr/mail/shared/decs:CONTROL=%h/Maildir/dovecot/public/control/decs:INDEX=%h/Maildir/dovecot/public/indexes/decs list: yes subscriptions: yes auth default: passdb: driver: pam userdb: driver: passwd args: system_user= socket: type: listen client: path: /var/spool/postfix/private/auth mode: 384 user: postfix group: postfix plugin: acl: vfile:/usr/local/etc/dovecot-acls:cache_secs=10 fts: squat
I think the latest http://dovecot.org/patches/1.1/listescape-plugin.c should work with latest Dovecot from hg. At least I fixed the most obvious problems.
Timo Sirainen wrote:
I think the latest http://dovecot.org/patches/1.1/listescape-plugin.c should work with latest Dovecot from hg. At least I fixed the most obvious problems.
Thats great to hear. Do you generally expect it to stay compatible with 1.1.x from now on? I'm in the beginning process of moving our users to dovecot and need to make careful choices on features I commit to supporting. Thanks.
On Feb 16, 2008, at 5:40 PM, Adam McDougall wrote:
Timo Sirainen wrote:
I think the latest http://dovecot.org/patches/1.1/listescape-plugin.c should work with latest Dovecot from hg. At least I fixed the most obvious problems.
Thats great to hear. Do you generally expect it to stay compatible
with 1.1.x from now on?
Yes.
Timo Sirainen wrote:
I think the latest http://dovecot.org/patches/1.1/listescape-plugin.c should work with latest Dovecot from hg. At least I fixed the most obvious problems.
I have been using it for a few weeks and it has been working well, but yesterday I noticed it also seems to be escaping the \ (backslash) character in folder names. This wouldn't be a problem except it breaks operation of folders that already had a \ in the name before the plugin was loaded. This isn't a huge problem for us but I want to figure out if I should special case migration of folder names containing a \ to be the new internal representation \5c or if the plugin should just deal with normal \'s in the foldernames properly.
Existing folder names with a \ such as \test still show up as \test in the mail client listing, but all known attempts to access it fail, I think internally it is probably looking for a folder named \5ctest and not finding it. Thus the user cannot read, delete, rename, unsubscribe, anything. They are stuck with a useless folder while the plugin is loaded. I doubt many of our users have this situation and they can contact us to fix it if they notice, but I did have one user alert me of it. On the other hand if a folder and its subscription are manually renamed to \5ctest, it works fine, or if they create a new one it uses a \5c on the backend.
Thanks for assistance.
"AM" == Adam McDougall <mcdouga9@egr.msu.edu> writes:
AM> I have been using it for a few weeks and it has been working well,
AM> but yesterday I noticed it also seems to be escaping the \
AM> (backslash) character in folder names.listescape has to escape '\' so that applications of decode-encode and encode-decode are idempotent. If it did not do this a client that asked, for example, to create a folder called a\2eb (i.e. issues CREATE "a\2eb") would be shown the folder as a.b by dovecot in a subsequent folder listing.
AM> This wouldn't be a problem except it breaks operation of folders
AM> that already had a \ in the name before the plugin was loaded.
AM> This isn't a huge problem for us but I want to figure out if I
AM> should special case migration of folder names containing a \ to be
AM> the new internal representation \5c or if the plugin should just
AM> deal with normal \'s in the foldernames properly.Yes, I think you need to rename folders that contained a '\' in the name before you started using listescape, but clearly you should not rename any folders that have subsequently been created by the listescape plugin that contain a '\'.
participants (3)
-
Adam McDougall
-
pod
-
Timo Sirainen