Changing IMAP separator - does it break things?
Dear Colleagues,
I am using Dovecot 2.3.4.1 with maildirs. Just found out that MacOS Mail clients cannot work with nested folders, presumably because the MacOS Mail does not understand "." as the IMAP separator.
Other clients (ThunderBird, Sylpheed, mutt) work fine.
Do you think I can change the hierarchy separator in the "inbox" namespace without breaking other clients and the mail layout on disk, losing mail etc?
I don't want to break mail for 50 people just to please one Mac Mail user. The users are advanced developers actively using mail sorting rules, nested folders etc.
My doveconf -n is here: https://termbin.com/w1wd
-- Victor Sudakov, VAS4-RIPE, VAS47-RIPN 2:5005/49@fidonet http://vas.tomsk.ru/
Hi Victor,
On 14.10.20 04:36, Victor Sudakov wrote:
Do you think I can change the hierarchy separator in the "inbox" namespace without breaking other clients and the mail layout on disk, losing mail etc?
I personally would try to avoid such a change (really IMHO), especially if it's "only":
I don't want to break mail for 50 people just to please one Mac Mail user.
_one_ user. Could Thunderbird or another IMAP client be an option for this single user? And perhaps Apple can tell some details when their Mail client will be standards-compliant?
Besides that: changing the hierarchy separator is generally possible. But you schould try and check that with a test system to prevent problems with your productive environment and all the other users.
Regards, Markus
- 2020 v 11:28, Markus Winkler <ml@irmawi.de>:
Hi Victor,
On 14.10.20 04:36, Victor Sudakov wrote:
Do you think I can change the hierarchy separator in the "inbox" namespace without breaking other clients and the mail layout on disk, losing mail etc?
I personally would try to avoid such a change (really IMHO), especially if it's "only":
I don't want to break mail for 50 people just to please one Mac Mail user.
_one_ user. Could Thunderbird or another IMAP client be an option for this single user? And perhaps Apple can tell some details when their Mail client will be standards-compliant?
Besides that: changing the hierarchy separator is generally possible. But you schould try and check that with a test system to prevent problems with your productive environment and all the other users.
I had to do it with 10-20 live users and it went fine, nobody noticed anything (I made the change in the night hours just to be sure though). I had to go down and fix some erratic folder names that resulted from the botched separator handling on some macOS Mail accounts.
The funny thing is that I set up a vanilla Dovecot server elsewhere and subfolders worked fine there on the very same macOS client. Couldn’t find any significant difference in the configs that would explain the problem, also the relative lack of similar reports out there seemed mind boggling.
(I’m using the Dovecot-based Mailcow mail stack for production, which makes troubleshooting a little more challenging.)
-F
Filip Hajný wrote:
On 14.10.20 04:36, Victor Sudakov wrote:
Do you think I can change the hierarchy separator in the "inbox" namespace without breaking other clients and the mail layout on disk, losing mail etc?
I personally would try to avoid such a change (really IMHO), especially if it's "only":
I don't want to break mail for 50 people just to please one Mac Mail user.
_one_ user. Could Thunderbird or another IMAP client be an option for this single user? And perhaps Apple can tell some details when their Mail client will be standards-compliant?
Besides that: changing the hierarchy separator is generally possible. But you schould try and check that with a test system to prevent problems with your productive environment and all the other users.
I had to do it with 10-20 live users and it went fine, nobody noticed anything (I made the change in the night hours just to be sure though). I had to go down and fix some erratic folder names that resulted from the botched separator handling on some macOS Mail accounts.
Did it change anything on Dovecot disk/storage or is the IMAP separator just "virtual" - ephemeral ?
The funny thing is that I set up a vanilla Dovecot server elsewhere and subfolders worked fine there on the very same macOS client. Couldn’t find any significant difference in the configs that would explain the problem, also the relative lack of similar reports out there seemed mind boggling.
Those Mac clients who do not use nested folders do not seem to be affected. That is, I've already suggested to the Mac Mail user that he start Thunderbird, move all his mailboxes into the IMAP root folder, and then go on using Mac Mail. I hope that's what he has done.
(I’m using the Dovecot-based Mailcow mail stack for production, which makes troubleshooting a little more challenging.)
-- Victor Sudakov, VAS4-RIPE, VAS47-RIPN 2:5005/49@fidonet http://vas.tomsk.ru/
- 2020 v 11:51, Victor Sudakov <vas@sibptus.ru>:
I had to do it with 10-20 live users and it went fine, nobody noticed anything (I made the change in the night hours just to be sure though). I had to go down and fix some erratic folder names that resulted from the botched separator handling on some macOS Mail accounts.
Did it change anything on Dovecot disk/storage or is the IMAP separator just "virtual" - ephemeral ?
Changing the separator won't fix anything that already exists. You should see nonsense folders in the form of '.folder\2esubfolder‘ there that cannot be worked with in any way. I manually moved the contents of these to their properly named ('.folder.subfolder‘) variants - on disk.
-F
Filip Hajný wrote:
I had to do it with 10-20 live users and it went fine, nobody noticed anything (I made the change in the night hours just to be sure though). I had to go down and fix some erratic folder names that resulted from the botched separator handling on some macOS Mail accounts.
Did it change anything on Dovecot disk/storage or is the IMAP separator just "virtual" - ephemeral ?
Changing the separator won't fix anything that already exists. You should see nonsense folders in the form of '.folder\2esubfolder‘ there that cannot be worked with in any way. I manually moved the contents of these to their properly named ('.folder.subfolder‘) variants - on disk.
Thanks for the warning Filip!
-- Victor Sudakov, VAS4-RIPE, VAS47-RIPN 2:5005/49@fidonet http://vas.tomsk.ru/
Filip Hajný wrote:
- 2020 v 11:51, Victor Sudakov <vas@sibptus.ru>:
I had to do it with 10-20 live users and it went fine, nobody noticed anything (I made the change in the night hours just to be sure though). I had to go down and fix some erratic folder names that resulted from the botched separator handling on some macOS Mail accounts.
Did it change anything on Dovecot disk/storage or is the IMAP separator just "virtual" - ephemeral ?
Changing the separator won't fix anything that already exists. You should see nonsense folders in the form of '.folder\2esubfolder‘ there that cannot be worked with in any way. I manually moved the contents of these to their properly named ('.folder.subfolder‘) variants - on disk.
I was asking because the section "Hierarchy separators" in https://doc.dovecot.org/configuration_manual/namespace/ somewhat implies that
However, changing the separator doesn’t change the on-disk “layout separator”.
That's why I thought that the separator is in fact virtual.
At present, if my folder layout in the mail client is
Inbox | +---Friends | +----Paul | +----Jessica
the folder "Jessica" on-disk is something like ~/Maildir/.Friends.Jessica/{cur,new,tmp}/
Do you mean to say that if I change the IMAP separator to "/", Dovecot will not translate "/" into "." and the folder "Jessica" will become inaccessible?
-- Victor Sudakov, VAS4-RIPE, VAS47-RIPN 2:5005/49@fidonet http://vas.tomsk.ru/
On 14.10.20 15:02, Victor Sudakov wrote:
I was asking because the section "Hierarchy separators" in https://doc.dovecot.org/configuration_manual/namespace/ somewhat implies that
However, changing the separator doesn’t change the on-disk “layout separator”.
That's why I thought that the separator is in fact virtual.
You're right - please see my former mail and the link you repeated above. There's a nice table which shows the possible combinations of hierarchy separator (called 'NS sep' there) and 'Layout sep' (structure on disk) and the respective results for 'Mailbox name' and the directory structure in the filesystem.
At present, if my folder layout in the mail client is
Inbox | +---Friends | +----Paul | +----Jessica
the folder "Jessica" on-disk is something like ~/Maildir/.Friends.Jessica/{cur,new,tmp}/ -------------------------------------------------^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Then you have configured a Layout sep '.' (mail_location = maildir:~/Maildir).
This says nothing about your hierarchy separator, which is controlled by the parameter 'separator ='.
Do you mean to say that if I change the IMAP separator to "/", Dovecot will not translate "/" into "." and the folder "Jessica" will become inaccessible?
If you currently have 'separator = .' then of course the hierarchy separator is '.'. If you change that to 'separator = /' normally the access to "Jessica" is still possible.
But (as you can see in the table mentioned above) the name of the mailbox (= folder) from the IMAP clients point of view is now 'Friends/Jessica'.
Before the change and with 'separator = .' the name of the mailbox was 'Friends.Jessica'.
If you telnet to Dovecot and (after authentication) type something like this:
a LIST "" "*"
The server may responds with:
- LIST (\HasChildren) "." INBOX
Here you can see, that this name space has a '.' as the hierarchy separator.
If the response is this:
- LIST (\HasChildren) "/" INBOX
then we know that '/' is the hierarchy separator.
So, if you have an IMAP client which ist standard-compliant he will check what's the configured hierarchy separator and will use it for accessing the folder structure. And again: this has nothing to do with the on disk structure of the folders.
Regards, Markus
On 14. Oct 2020, at 12.51, Victor Sudakov <vas@sibptus.ru> wrote:
Those Mac clients who do not use nested folders do not seem to be affected. That is, I've already suggested to the Mac Mail user that he start Thunderbird, move all his mailboxes into the IMAP root folder, and then go on using Mac Mail. I hope that's what he has done.
There is a bug in Mail.app that it always assumes / as hierarchy separator and completely ignores server provided separator.
Not sure if anyone has created BUG to Apple about this. Atleast I have not.
Sami
- 2020 v 12:24, Sami Ketola <sami@ketola.io>:
On 14. Oct 2020, at 12.51, Victor Sudakov <vas@sibptus.ru> wrote:
Those Mac clients who do not use nested folders do not seem to be affected. That is, I've already suggested to the Mac Mail user that he start Thunderbird, move all his mailboxes into the IMAP root folder, and then go on using Mac Mail. I hope that's what he has done.
There is a bug in Mail.app that it always assumes / as hierarchy separator and completely ignores server provided separator.
Not sure if anyone has created BUG to Apple about this. Atleast I have not.
Sami
So hm, I actually re-read the original post, and this is pretty weird, because I had to change the Dovecot separator from '/' to '.' in order to make macOS clients work with subfolders properly.
I never had this problem myself because only use folders, not subfolders. And it was pointed to me by another user, who relies on subfolders and had problems for some time - *maybe* from since around the time she upgraded to Catalina.
-F
On 14 Oct 2020, at 04:24, Sami Ketola <sami@ketola.io> wrote:
On 14. Oct 2020, at 12.51, Victor Sudakov <vas@sibptus.ru> wrote:
Those Mac clients who do not use nested folders do not seem to be affected. That is, I've already suggested to the Mac Mail user that he start Thunderbird, move all his mailboxes into the IMAP root folder, and then go on using Mac Mail. I hope that's what he has done.
There is a bug in Mail.app that it always assumes / as hierarchy separator and completely ignores server provided separator.
Cite? Because no, that is not my experience at all.
From my maildir on my FreeBSD server:
# ls -lnd .root* drwx------ 5 89 89 512 Oct 14 05:16 .root drwx------ 5 89 89 512 Oct 14 02:02 .root.44 drwx------ 5 89 89 512 Oct 14 02:02 .root.Cron drwx------ 5 89 89 512 Oct 14 05:16 .root.Hastur … etc
Works fine in Mail.app on Mac OS and has for at least 16 years.
-- I'm not old, I'm chronologically challenged.
On 14. Oct 2020, at 14.20, @lbutlr <kremels@kreme.com> wrote:
On 14 Oct 2020, at 04:24, Sami Ketola <sami@ketola.io> wrote:
On 14. Oct 2020, at 12.51, Victor Sudakov <vas@sibptus.ru> wrote:
Those Mac clients who do not use nested folders do not seem to be affected. That is, I've already suggested to the Mac Mail user that he start Thunderbird, move all his mailboxes into the IMAP root folder, and then go on using Mac Mail. I hope that's what he has done.
There is a bug in Mail.app that it always assumes / as hierarchy separator and completely ignores server provided separator.
Cite? Because no, that is not my experience at all.
From my maildir on my FreeBSD server:
# ls -lnd .root* drwx------ 5 89 89 512 Oct 14 05:16 .root drwx------ 5 89 89 512 Oct 14 02:02 .root.44 drwx------ 5 89 89 512 Oct 14 02:02 .root.Cron drwx------ 5 89 89 512 Oct 14 05:16 .root.Hastur … etc
Works fine in Mail.app on Mac OS and has for at least 16 years.
I just tested it with my Dovecot 2.2.36 setup. I did set up an account with '.' as separator and created some folders. First mail.app succeeded in creating test root folder but when I tried to create subfolder 'test2' under 'test' Mail.app issued CREATE "test/test2" folder instead of "test.test2". And then failed to list the created folder completely.
Everything worked if I did set separator to '/'
Sami
- 2020 v 13:39, Sami Ketola <sami@ketola.io>:
On 14. Oct 2020, at 14.20, @lbutlr <kremels@kreme.com> wrote:
On 14 Oct 2020, at 04:24, Sami Ketola <sami@ketola.io> wrote:
On 14. Oct 2020, at 12.51, Victor Sudakov <vas@sibptus.ru> wrote:
Those Mac clients who do not use nested folders do not seem to be affected. That is, I've already suggested to the Mac Mail user that he start Thunderbird, move all his mailboxes into the IMAP root folder, and then go on using Mac Mail. I hope that's what he has done.
There is a bug in Mail.app that it always assumes / as hierarchy separator and completely ignores server provided separator.
Cite? Because no, that is not my experience at all.
From my maildir on my FreeBSD server:
# ls -lnd .root* drwx------ 5 89 89 512 Oct 14 05:16 .root drwx------ 5 89 89 512 Oct 14 02:02 .root.44 drwx------ 5 89 89 512 Oct 14 02:02 .root.Cron drwx------ 5 89 89 512 Oct 14 05:16 .root.Hastur … etc
Works fine in Mail.app on Mac OS and has for at least 16 years.
I just tested it with my Dovecot 2.2.36 setup. I did set up an account with '.' as separator and created some folders. First mail.app succeeded in creating test root folder but when I tried to create subfolder 'test2' under 'test' Mail.app issued CREATE "test/test2" folder instead of "test.test2". And then failed to list the created folder completely.
Everything worked if I did set separator to '/'
In my case, having the hierarchy separator set to '/', subfolders created using the Mail client in macOS Catalina resulted in '.folder\2esubfolder‘ on disk (2e corresponds to '.‘). I had to change the hierarchy separator to a dot.
So exactly the opposite!
-F
On 14 Oct 2020, at 05:39, Sami Ketola <sami@ketola.io> wrote:
On 14. Oct 2020, at 14.20, @lbutlr <kremels@kreme.com> wrote:
On 14 Oct 2020, at 04:24, Sami Ketola <sami@ketola.io> wrote:
On 14. Oct 2020, at 12.51, Victor Sudakov <vas@sibptus.ru> wrote:
Those Mac clients who do not use nested folders do not seem to be affected. That is, I've already suggested to the Mac Mail user that he start Thunderbird, move all his mailboxes into the IMAP root folder, and then go on using Mac Mail. I hope that's what he has done.
There is a bug in Mail.app that it always assumes / as hierarchy separator and completely ignores server provided separator.
Cite? Because no, that is not my experience at all.
From my maildir on my FreeBSD server:
# ls -lnd .root* drwx------ 5 89 89 512 Oct 14 05:16 .root drwx------ 5 89 89 512 Oct 14 02:02 .root.44 drwx------ 5 89 89 512 Oct 14 02:02 .root.Cron drwx------ 5 89 89 512 Oct 14 05:16 .root.Hastur … etc
Works fine in Mail.app on Mac OS and has for at least 16 years.
I just tested it with my Dovecot 2.2.36 setup. I did set up an account with '.' as separator and created some folders.
Are you ignoring the advice in Dovecot to leave its jeorachu separator field alone?
There's a difference between the IMAP separator and the hierarchy separator, and you should not change (or set) the latter at all.
-- Oh, he's just like any other man, only more so.
@lbutlr wrote:
On 14 Oct 2020, at 04:24, Sami Ketola <sami@ketola.io> wrote:
On 14. Oct 2020, at 12.51, Victor Sudakov <vas@sibptus.ru> wrote:
Those Mac clients who do not use nested folders do not seem to be affected. That is, I've already suggested to the Mac Mail user that he start Thunderbird, move all his mailboxes into the IMAP root folder, and then go on using Mac Mail. I hope that's what he has done.
There is a bug in Mail.app that it always assumes / as hierarchy separator and completely ignores server provided separator.
Cite? Because no, that is not my experience at all.
Cannot site a bug, can only site a users's complaint: https://discussions.apple.com/thread/8347018
From my maildir on my FreeBSD server:
# ls -lnd .root* drwx------ 5 89 89 512 Oct 14 05:16 .root drwx------ 5 89 89 512 Oct 14 02:02 .root.44 drwx------ 5 89 89 512 Oct 14 02:02 .root.Cron drwx------ 5 89 89 512 Oct 14 05:16 .root.Hastur … etc
Works fine in Mail.app on Mac OS and has for at least 16 years.
-- Victor Sudakov, VAS4-RIPE, VAS47-RIPN 2:5005/49@fidonet http://vas.tomsk.ru/
Hi Victor,
On 14.10.20 11:51, Victor Sudakov wrote:
Did it change anything on Dovecot disk/storage or is the IMAP separator just "virtual" - ephemeral ?
just for the for the sake of completeness: there is a difference between the hierarchy separator (for the mailbox name) and the layout separator, which is responsible for the folder names on the disk - they are completely independent.
Here you can see the details, if you not already know this page:
https://doc.dovecot.org/configuration_manual/namespace/
Regards, Markus
Markus Winkler wrote:
On 14.10.20 04:36, Victor Sudakov wrote:
Do you think I can change the hierarchy separator in the "inbox" namespace without breaking other clients and the mail layout on disk, losing mail etc?
I personally would try to avoid such a change (really IMHO), especially if it's "only":
I don't want to break mail for 50 people just to please one Mac Mail user.
_one_ user. Could Thunderbird or another IMAP client be an option for this single user? And perhaps Apple can tell some details when their Mail client will be standards-compliant?
Looking at https://discussions.apple.com/thread/8347018 I don't hold my breath.
Besides that: changing the hierarchy separator is generally possible. But you schould try and check that with a test system to prevent problems with your productive environment and all the other users.
I was hoping someone had already done that and I could skip this step :-)
-- Victor Sudakov, VAS4-RIPE, VAS47-RIPN 2:5005/49@fidonet http://vas.tomsk.ru/
On 14.10.20 11:57, Victor Sudakov wrote:
Besides that: changing the hierarchy separator is generally possible. But you schould try and check that with a test system to prevent problems with your productive environment and all the other users.
I was hoping someone had already done that and I could skip this step :-)
No, I didn't. ;-) I almost always use '/' as the separator so I didn't observed any problem with Mac Mail users.
Regards, Markus
On 13 Oct 2020, at 20:36, Victor Sudakov <vas@sibptus.ru> wrote:
I am using Dovecot 2.3.4.1 with maildirs. Just found out that MacOS Mail clients cannot work with nested folders, presumably because the MacOS Mail does not understand "." as the IMAP separator.
This is not correct. Mac Mail has no problems with nested folders and never has had any problem with nested folders. I have always used the '.' separator and I've had a 'Root' mail folder with subfolders since at least 2004.
I don't want to break mail for 50 people just to please one Mac Mail user. The users are advanced developers actively using mail sorting rules, nested folders etc.
And what makes them think nested folders do not work?
-- Alice: If you would just shut up for about for two seconds, this sex dream would pass the Bechdel test.
@lbutlr wrote:
On 13 Oct 2020, at 20:36, Victor Sudakov <vas@sibptus.ru> wrote:
I am using Dovecot 2.3.4.1 with maildirs. Just found out that MacOS Mail clients cannot work with nested folders, presumably because the MacOS Mail does not understand "." as the IMAP separator.
This is not correct. Mac Mail has no problems with nested folders and never has had any problem with nested folders. I have always used the '.' separator and I've had a 'Root' mail folder with subfolders since at least 2004.
I think I have already posted this link: https://discussions.apple.com/thread/8347018
Also this: http://hints.macworld.com/article.php?story=2007090423071517 It contains weird advice but at least proves that the problem exists for *some* people. See also the comments therein.
I don't want to break mail for 50 people just to please one Mac Mail user. The users are advanced developers actively using mail sorting rules, nested folders etc.
And what makes them think nested folders do not work?
I don't have the exact error message from him but it was pretty obvious about a folder "foo/bar/bob" being non-existent or inaccessible. And indeed, it's "~/Maildir/.foo.bar.bob" on the IMAP server's disk, there are no subdirectories for mail folders.
-- Victor Sudakov, VAS4-RIPE, VAS47-RIPN 2:5005/49@fidonet http://vas.tomsk.ru/
participants (5)
-
＠lbutlr
-
Filip Hajný
-
Markus Winkler
-
Sami Ketola
-
Victor Sudakov