[Dovecot] . (dot) in maildir folder names
Hi.
Uhm... yeah as the topic implies I want to have . (dots) in my folder names... Unfortunately dovecot's maildirmake hasn't a -f switch as the one from courier/maildrop, but that one in turn is buggy[0] and doesn't encode any characters at all.
In principle, maildir++ should allow[1] having (encoded) dots in folder names,... but I don't quite understand how the encoding actually works, cause UTF-7 seems to be already the encoded (without padding and such)... but . is . in UTF-7 as well.
Does dovecot support this? At least my mail clients (well I only tried Evolution so far which sucks in so many areas)... couldn't create it... but I guess it's rather a stupid client issue. Anyone knows how it would be encoded?
Interestingly, and this might be actually an inssue in doveocot when I create maildirs with non-ASCII chars without encoding these, e.g. maildir/.München/ (instead of the correctly encoded (.M&APw-nchen) Dovecot seems to "export" these to the clients,... at least Evolution shows it in the list of subscribable folders (even correctly as "München")... but it doesn't allow me to actually subscribe to it.
Cheers & thx, Chris.
[0] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=709555 [1] http://www.courier-mta.org/maildir.html#id351803
Maybe the whole thing IS actually also a dovecot issue... When trying to create a folder with a dot with thunderbird (which doesn't simply forbid this)... it actually transmits the folder name to dovecote (e.g. "kernel.org")... but dovecot then literally takes this over and creates .kernel.org, i.e. no encoding and thus org becomes a subfolder.
Cheers, Chris.
On 2013-05-23 10:12 PM, Christoph Anton Mitterer calestyo@scientia.net wrote:
Maybe the whole thing IS actually also a dovecot issue... When trying to create a folder with a dot with thunderbird (which doesn't simply forbid this)... it actually transmits the folder name to dovecote (e.g. "kernel.org")... but dovecot then literally takes this over and creates .kernel.org, i.e. no encoding and thus org becomes a subfolder.
And for some reason, the Parent folder is created as a folder that cannot contain files, only folders.
I have to fix these occasionally for users who do this.
This is why I'll be implementing a different character for the separator on my new box - something much less likely to be used in a foldername - I'm currently thinking the caret (^) symbol may be best...
--
Best regards,
Charles
On Fri, 2013-05-24 at 06:57 -0400, Charles Marcus wrote:
And for some reason, the Parent folder is created as a folder that cannot contain files, only folders. I'll try that out..
This is why I'll be implementing a different character for the separator on my new box - something much less likely to be used in a foldername - I'm currently thinking the caret (^) symbol may be best... Sure.... but it's not a real solution... it only moves the problem.
Cheers, Chris.
On 2013-05-24 8:32 AM, Christoph Anton Mitterer calestyo@scientia.net wrote:
On Fri, 2013-05-24 at 06:57 -0400, Charles Marcus wrote:
This is why I'll be implementing a different character for the separator on my new box - something much less likely to be used in a foldername - I'm currently thinking the caret (^) symbol may be best... Sure.... but it's not a real solution... it only moves the problem.
Well... I disagree.
If changing the separator to a character that 99.999% of the time no one will ever use in a foldername, it eliminates the problem in 99.999% of cases.
I can live with that... ;)
--
Best regards,
Charles
On Fri, 2013-05-24 at 06:57 -0400, Charles Marcus wrote:
And for some reason, the Parent folder is created as a folder that cannot contain files, only folders. I think the reason for this is that the folder is not really created, i.e. when you create "folder.subfolder" with TB, Dovcote will create .folder.subfolder
But there is not maildir for: .folder
Not sure whom to blame for this... is it a dovecot issue?
Cheers, Chris.
On 2013-05-23 18:41, Christoph Anton Mitterer wrote:
Uhm... yeah as the topic implies I want to have . (dots) in my folder names... Unfortunately dovecot's maildirmake hasn't a -f switch as the one from courier/maildrop, but that one in turn is buggy[0] and doesn't encode any characters at all.
Does adding LAYOUT=fs to mail_location, which makes Dovecot use a dir hierarchy instead of dot-prefixing, make this possible?
P.S., You don't need to use a maildirmake script with Dovecot--it will automatically create the Maildir on the first mail delivery (assuming you're using the deliver agent).
-- Please reply on list.
On Thu, 2013-05-23 at 19:58 -0700, Darren Pilgrim wrote:
Does adding LAYOUT=fs to mail_location, which makes Dovecot use a dir hierarchy instead of dot-prefixing, make this possible? I would expect that one then runs into the same troubles when using "/" in a foldername... Moreover one has the problem that cur/new/tmp need to be hidden in another special dir name that mustn't be used.
Cheers, Chris.
On 2013-05-24 05:29, Christoph Anton Mitterer wrote:
On Thu, 2013-05-23 at 19:58 -0700, Darren Pilgrim wrote:
Does adding LAYOUT=fs to mail_location, which makes Dovecot use a dir hierarchy instead of dot-prefixing, make this possible?
I would expect that one then runs into the same troubles when using "/" in a foldername...
I often wondered by '.' was chosen. A comma works just as well and, unlike '.', ',' isn't valid in a domain name or unquoted local part.
Using the FS does solve the only major issue I run into doing automated mailing list sorting based on the List-ID header.
Am 24.05.2013 03:41, schrieb Christoph Anton Mitterer:
Hi.
Uhm... yeah as the topic implies I want to have . (dots) in my folder names... Unfortunately dovecot's maildirmake hasn't a -f switch as the one from courier/maildrop, but that one in turn is buggy[0] and doesn't encode any characters at all.
In principle, maildir++ should allow[1] having (encoded) dots in folder names,... but I don't quite understand how the encoding actually works, cause UTF-7 seems to be already the encoded (without padding and such)... but . is . in UTF-7 as well.
Does dovecot support this? At least my mail clients (well I only tried Evolution so far which sucks in so many areas)... couldn't create it... but I guess it's rather a stupid client issue. Anyone knows how it would be encoded?
Interestingly, and this might be actually an inssue in doveocot when I create maildirs with non-ASCII chars without encoding these, e.g. maildir/.München/ (instead of the correctly encoded (.M&APw-nchen) Dovecot seems to "export" these to the clients,... at least Evolution shows it in the list of subscribable folders (even correctly as "München")... but it doesn't allow me to actually subscribe to it.
Cheers & thx, Chris.
[0] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=709555 [1] http://www.courier-mta.org/maildir.html#id351803
look
http://wiki2.dovecot.org/Plugins/Listescape
Best Regards MfG Robert Schetterer
-- [*] sys4 AG
http://sys4.de, +49 (89) 30 90 46 64 Franziskanerstraße 15, 81669 München
Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263 Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer Aufsichtsratsvorsitzender: Florian Kirstein
http://wiki2.dovecot.org/Plugins/Listescape That sounds interesting... and I'll have a look at it.. but it also seems to use an standard-incompliant encoding (\NN)... might become a
On Fri, 2013-05-24 at 08:32 +0200, Robert Schetterer wrote: problem when one uses other tools on such maildirs (maildrop?)
Cheers, Chris.
On 24.5.2013, at 15.31, Christoph Anton Mitterer calestyo@scientia.net wrote:
http://wiki2.dovecot.org/Plugins/Listescape That sounds interesting... and I'll have a look at it.. but it also seems to use an standard-incompliant encoding (\NN)... might become a
On Fri, 2013-05-24 at 08:32 +0200, Robert Schetterer wrote: problem when one uses other tools on such maildirs (maildrop?)
There is no standard escaping.
Well... I thought courier was defining maildir++ which we use an they also defined an escaping....
Cheers, Chris.
Timo Sirainen tss@iki.fi schrieb:
On 24.5.2013, at 15.31, Christoph Anton Mitterer calestyo@scientia.net wrote:
http://wiki2.dovecot.org/Plugins/Listescape That sounds interesting... and I'll have a look at it.. but it also seems to use an standard-incompliant encoding (\NN)... might become a
On Fri, 2013-05-24 at 08:32 +0200, Robert Schetterer wrote: problem when one uses other tools on such maildirs (maildrop?)
There is no standard escaping.
I'm not aware of Courier doing any kind of escaping.
On 24.5.2013, at 17.08, Christoph Anton Mitterer calestyo@scientia.net wrote:
Well... I thought courier was defining maildir++ which we use an they also defined an escaping....
Cheers, Chris.
Timo Sirainen tss@iki.fi schrieb:
On 24.5.2013, at 15.31, Christoph Anton Mitterer calestyo@scientia.net wrote:
http://wiki2.dovecot.org/Plugins/Listescape That sounds interesting... and I'll have a look at it.. but it also seems to use an standard-incompliant encoding (\NN)... might become a
On Fri, 2013-05-24 at 08:32 +0200, Robert Schetterer wrote: problem when one uses other tools on such maildirs (maildrop?)
There is no standard escaping.
Am 24.05.2013 16:08, schrieb Christoph Anton Mitterer:
Well... I thought courier was defining maildir++ which we use an they also defined an escaping....
Cheers, Chris.
Timo Sirainen tss@iki.fi schrieb:
On 24.5.2013, at 15.31, Christoph Anton Mitterer calestyo@scientia.net wrote:
http://wiki2.dovecot.org/Plugins/Listescape That sounds interesting... and I'll have a look at it.. but it also seems to use an standard-incompliant encoding (\NN)... might become a
On Fri, 2013-05-24 at 08:32 +0200, Robert Schetterer wrote: problem when one uses other tools on such maildirs (maildrop?)
There is no standard escaping.
however, just for Info, i use Listescape with maildir in 2.1.15 without Problems, i used it for archive mailbox with sieve sorting in subfolders named like email adresses ( which might have a dot )
Best Regards MfG Robert Schetterer
-- [*] sys4 AG
http://sys4.de, +49 (89) 30 90 46 64 Franziskanerstraße 15, 81669 München
Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263 Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer Aufsichtsratsvorsitzender: Florian Kirstein
On Fri, 2013-05-24 at 16:14 +0300, Timo Sirainen wrote:
There is no standard escaping. btw... When I create a folder with unicode chars ... e.g. via Evolution or TB.. then the maildirs created on the dovecot side use the encoding schema as described here http://www.courier-mta.org/maildir.html
Is this encoding then done by Dovecot or is it the client who does this?
Chris.
On 25.5.2013, at 0.48, Christoph Anton Mitterer calestyo@scientia.net wrote:
On Fri, 2013-05-24 at 16:14 +0300, Timo Sirainen wrote:
There is no standard escaping. btw... When I create a folder with unicode chars ... e.g. via Evolution or TB.. then the maildirs created on the dovecot side use the encoding schema as described here http://www.courier-mta.org/maildir.html
Is this encoding then done by Dovecot or is it the client who does this?
That is pretty much standard IMAP mUTF-7. Wonder why that's not mentioned in there.
But yeah, looks like it also suggests encoding the '.' and '/' using the mUTF-7 format. That is forbidden by IMAP. So having such names in the maildir and serving them to IMAP clients would actually violate the IMAP protocol.
On Sat, 2013-05-25 at 01:19 +0300, Timo Sirainen wrote:
That is pretty much standard IMAP mUTF-7. Wonder why that's not mentioned in there. But yeah, looks like it also suggests encoding the '.' and '/' using the mUTF-7 format. That is forbidden by IMAP. So having such names in the maildir and serving them to IMAP clients would actually violate the IMAP protocol. I see... so what's the best / most standards compliant way of enabling all possible folder names now? Using that plugin?
I mean I do not quite understand what happens when I use the plugin... so as far as I understood you know IMAP itself already has an encoding way... i.e. the client already sends mUTF-7 encoded foldernames to dovecot, and dovecot simply passes this through and creates these as file names, right?
When I use the list encode plugin... will it then \NN encode the (possibly already mUTF-7 encoded) string... to also allow things like "." and "/"? So the client will never see any \NN encoding as this is done purely dovecot internally?
Thanks, Chris.
On 25.5.2013, at 1.46, Christoph Anton Mitterer calestyo@scientia.net wrote:
On Sat, 2013-05-25 at 01:19 +0300, Timo Sirainen wrote:
That is pretty much standard IMAP mUTF-7. Wonder why that's not mentioned in there. But yeah, looks like it also suggests encoding the '.' and '/' using the mUTF-7 format. That is forbidden by IMAP. So having such names in the maildir and serving them to IMAP clients would actually violate the IMAP protocol. I see... so what's the best / most standards compliant way of enabling all possible folder names now? Using that plugin?
IMAP protocol requires that one character is reserved for being hierarchy separator. There's no way around that. But with listescape plugin you can use any other character.
I mean I do not quite understand what happens when I use the plugin... so as far as I understood you know IMAP itself already has an encoding way... i.e. the client already sends mUTF-7 encoded foldernames to dovecot, and dovecot simply passes this through and creates these as file names, right?
When I use the list encode plugin... will it then \NN encode the (possibly already mUTF-7 encoded) string... to also allow things like "." and "/"? So the client will never see any \NN encoding as this is done purely dovecot internally?
Right. It encodes those chars that can't be used in filesystem, but can be used in IMAP protocol. With Maildir the separator is always '.' in the filesystem, but it can be something else visible to IMAP clients, which allows using '.' in IMAP protocol but not in filesystem unescaped.
On Sat, 2013-05-25 at 02:12 +0300, Timo Sirainen wrote:
IMAP protocol requires that one character is reserved for being hierarchy separator. There's no way around that. But with listescape plugin you can use any other character. Okay... than IMAP really sucks in that matter ^^... why not allowing it to be used quoted ... weird...
IIRC, clients can find out that character via the LIST command, right? So there should be no need for them to block anything else but that specific character (per server).
Right. It encodes those chars that can't be used in filesystem, but can be used in IMAP protocol. With Maildir the separator is always '.' in the filesystem well unless one use LAYOUT=fs
, but it can be something else visible to IMAP clients, which allows using '.' in IMAP protocol but not in filesystem unescaped.
but that basically also means that the plugins will never encode characters but . and / (okay and that ~ case, which AFAIU wouldn't be strictly necessary), right?
It will encode "/" because it can't be used in the POSIX filenames. And it will encode "." only when LAYOUT=fs is _not_ used. Other Unicode chars are already encoded by IMAP via UTF7 and can go straight into the FS.
And the virtual separator cannot be used, regardless what one does... Is this just an IMAP issue or also a POP issue?
And how are other unicode chars encoded in POP? I ask cause then I'd update the wiki a bit.
Cheers, Chris.
On 25.5.2013, at 2.30, Christoph Anton Mitterer calestyo@scientia.net wrote:
On Sat, 2013-05-25 at 02:12 +0300, Timo Sirainen wrote:
IMAP protocol requires that one character is reserved for being hierarchy separator. There's no way around that. But with listescape plugin you can use any other character. Okay... than IMAP really sucks in that matter ^^... why not allowing it to be used quoted ... weird...
IIRC, clients can find out that character via the LIST command, right? So there should be no need for them to block anything else but that specific character (per server).
Yeah.
, but it can be something else visible to IMAP clients, which allows using '.' in IMAP protocol but not in filesystem unescaped.
but that basically also means that the plugins will never encode characters but . and / (okay and that ~ case, which AFAIU wouldn't be strictly necessary), right?
It will encode "/" because it can't be used in the POSIX filenames. And it will encode "." only when LAYOUT=fs is _not_ used. Other Unicode chars are already encoded by IMAP via UTF7 and can go straight into the FS.
Right.
And the virtual separator cannot be used, regardless what one does... Is this just an IMAP issue or also a POP issue?
And how are other unicode chars encoded in POP? I ask cause then I'd update the wiki a bit.
POP3 has no folders at all.
And the virtual separator cannot be used, regardless what one does... Is this just an IMAP issue or also a POP issue?
And how are other unicode chars encoded in POP? I ask cause then I'd update the wiki a bit.
POP3 has no folders at all. Ah sure... I didn't think clearly.... what I actually meant was... are
On Sat, 2013-05-25 at 02:33 +0300, Timo Sirainen wrote: there similar issues when using folder with mbox as backend? I mean I haven't tried it with Dovecot so far, but I guess it also uses the folder.sbd/ schema? Does this also involve any special forbidden characters?
I'm just making some tests... and everything seems to be as you described (not that I didn't trust you ;) )...
One thing though... I think when a character is used that is neither the (IMAP) virtual separator (and therefore already forbidden as a folder character by this
- which a good client could check[0]) nor the listencode plugin is enabled... then Dovecot should give an error when one tries to create a folder that contains a character special to the underlying mail storage.
Example: separator = / listencode is disabled maildir is used withOUT LAYOUT=fs
I create a folder named "foo.bar" => The client cannot know that "." causes troubles as it only knows about "/". Dovecot actually creates such a folder but it's the issue described before by some person here, that you have a non working folder "foo" and a subfolder "bar".
I think, Dovecot should bail out in such a folder creation, if that is possible. Especially as it seems one cannot get rid of such folders anymore... Evolution e.g. still thinks there should be a folder "foo" and it leaves it even when I deleted "bar". And one could make perhaps even more evil things like creating folders named "..." (and maildir get's crazy ;) )
I made some changes to the wiki,... guess you got notified anyway... please double check them.
Cheers, Chris
[0] To my big surprise.... evolution does... and e.g. allows "." if "/" is used as separator :-O
And I may have found one further issue:
I set: separator = '\\' maildir with _not_ using LAYOUT=fs the list encode plugin is on listescape_char is left to default
When I now create a folder "foo.bar" it seemingly works (the client shows it as "a.a"), but what comes out is: .a.2ea how can this work?
Chris.
On Sat, 2013-05-25 at 02:43 +0200, Christoph Anton Mitterer wrote:
And I may have found one further issue:
I set: separator = '\\' maildir with _not_ using LAYOUT=fs the list encode plugin is on listescape_char is left to default
When I now create a folder "foo.bar" it seemingly works (the client shows it as "a.a"), but what comes out is: .a.2ea how can this work?
Quite sure this must be a bug... I played a bit more... I can create a folder which is shown as "a.2ea" and in the filesystem this becomes ".a.2e2ea".
When I make a folder "a" and then try to make a subfolder "2ea" of it... it doesn't work (the client seems to already fail).
When I make subfolders (below) like "3e" or "65"... they first show up as these.... but after several times refreshing... they become folder "a>" and "ae".
Using another quote character like: listescape_char = ^ and things seem to work again as expected, i.e. a folder "e.e" becomes ".e^2ee"
Cheers, Chris.
Took a while, but finally fixed :) http://hg.dovecot.org/dovecot-2.2/rev/fce84463f508
On 25.5.2013, at 5.21, Christoph Anton Mitterer calestyo@scientia.net wrote:
On Sat, 2013-05-25 at 02:43 +0200, Christoph Anton Mitterer wrote:
And I may have found one further issue:
I set: separator = '\\' maildir with _not_ using LAYOUT=fs the list encode plugin is on listescape_char is left to default
When I now create a folder "foo.bar" it seemingly works (the client shows it as "a.a"), but what comes out is: .a.2ea how can this work?
Quite sure this must be a bug... I played a bit more... I can create a folder which is shown as "a.2ea" and in the filesystem this becomes ".a.2e2ea".
When I make a folder "a" and then try to make a subfolder "2ea" of it... it doesn't work (the client seems to already fail).
When I make subfolders (below) like "3e" or "65"... they first show up as these.... but after several times refreshing... they become folder "a>" and "ae".
Using another quote character like: listescape_char = ^ and things seem to work again as expected, i.e. a folder "e.e" becomes ".e^2ee"
Cheers, Chris.
participants (5)
-
Charles Marcus
-
Christoph Anton Mitterer
-
Darren Pilgrim
-
Robert Schetterer
-
Timo Sirainen