[Dovecot] 2.0.12 (fa2e92526716): Deleting mails and folders from public namespaces
Hello,
we basically run a dovecot imap setup where users' mailboxes are in private namespaces (as usual), but there is a public namespace with several thousand deeply nested folders also.
The client we use is Thunderbird 3. The public namespace shows up in each user's account at the top level (i.e. same level as Inbox, Sent and so on), and users can make new subfolders in the public namespace, copy mails to it and so on. So, our setup basically works.
But whenever a user tries to delete a folder from the public namespace, dovecot gives the following error: "Renaming not supported across non-private namespaces".
I think I have a grasp why this happens (Thunderbird tries to move the folder from the public namespace into the trash folder (private namespace) of the user who does the deletion), and I also have read Timo's statement that he does not plan to support such renaming due to big problems he is seeing with it.
Nevertheless, since deleting a folder from a public archive is not a too uncommon thing, I wanted to ask how other people have solved that.
It came to my mind to configure Thunderbird to immediately delete folders and messages (instead of moving them to trash folder). I have tested that, and it worked, but is somehow dangerous and unsatisfying.
Furthermore, I haven't completely understood the reason for the different behaviour of folders in the public namespace and a single message therein.
If I configure Thunderbird to use the trash folder, users can delete messages from the public namespace, i.e. Thunderbird can move messages from the public namespace to the private trash folder of the respective account.
This is obviously not true for folders (even if they are empty). What's the difference between an empty folder and a single message in this respect? Is it really the ACL thing which makes folders so difficult to handle opposed to single messages?
Thank you very much for any ideas!
Regards,
Peter
dovecot -n:
# 2.0.12 (fa2e92526716): /etc/dovecot/dovecot.conf # OS: Linux 2.6.32-5-amd64 x86_64 Debian 6.0.1 auth_verbose = yes auth_verbose_passwords = sha1 first_valid_uid = 103 maildir_copy_with_hardlinks = no namespace { hidden = no inbox = yes list = yes location = maildir:/home/vmail/content/users/%u:INDEX=/home/vmail/index/users/%u prefix = separator = / subscriptions = yes type = private } namespace { hidden = no inbox = no list = yes location = maildir:/home/vmail/content/archive:INDEX=/home/vmail/index/archive prefix = Archive/ separator = / subscriptions = yes type = public } passdb { args = scheme=SSHA username_format=%u /etc/dovecot/users driver = passwd-file } plugin { acl = vfile } protocols = " imap" service auth { user = root } service imap-login { inet_listener imap { port = 0 } inet_listener imaps { address = 192.168.20.140 port = 993 ssl = yes } } ssl_cert =
On 26.4.2011, at 23.48, Binarus wrote:
But whenever a user tries to delete a folder from the public namespace, dovecot gives the following error: "Renaming not supported across non-private namespaces". .. Furthermore, I haven't completely understood the reason for the different behaviour of folders in the public namespace and a single message therein.
Talking only about ACLs I guess it would be ok to allow renaming from public namespace to a private namespace. Renaming one of your own private mailboxes to public namespace is a bit trickier, because the default ACL is "no permissions". So when renaming, a new explicit ACL should be added to the user who did the renaming, because otherwise user could no longer access that mailbox at all. Still doable I guess.
Although it is possible to give user permission to delete foo/bar but not give permission to create children to foo. Then if user renames foo/bar to Trash/foo/bar, it's not possible to rename it back. Probably not a real problem.
Renaming from/to shared namespace needs to update the shared users acl dict.
Quota also needs to be updated. I'm not sure if lazy-expunge has any trouble with this. Multi-dbox backend can't do a fast rename at all, because it would have to physically copy all of the messages from one mdbox storage to another mdbox storage. Maybe there are other problems I can't remember now.
So looks like it should be possible to do all this, with the exception of adding an inefficiency to mdbox, but it's some work..
On 26.04.2011 23:25, Timo Sirainen wrote:
On 26.4.2011, at 23.48, Binarus wrote:
But whenever a user tries to delete a folder from the public namespace, dovecot gives the following error: "Renaming not supported across non-private namespaces". .. Furthermore, I haven't completely understood the reason for the different behaviour of folders in the public namespace and a single message therein.
Talking only about ACLs I guess it would be ok to allow renaming from public namespace to a private namespace. Renaming one of your own private mailboxes to public namespace is a bit trickier, because the default ACL is "no permissions". So when renaming, a new explicit ACL should be added to the user who did the renaming, because otherwise user could no longer access that mailbox at all. Still doable I guess.
Although it is possible to give user permission to delete foo/bar but not give permission to create children to foo. Then if user renames foo/bar to Trash/foo/bar, it's not possible to rename it back. Probably not a real problem.
Renaming from/to shared namespace needs to update the shared users acl dict.
Quota also needs to be updated. I'm not sure if lazy-expunge has any trouble with this. Multi-dbox backend can't do a fast rename at all, because it would have to physically copy all of the messages from one mdbox storage to another mdbox storage. Maybe there are other problems I can't remember now.
So looks like it should be possible to do all this, with the exception of adding an inefficiency to mdbox, but it's some work..
Thank you very much for your answer which shows that -as always- problems are more complicated than they seem first.
Nevertheless, I don't want to give up, have a few thoughts and just would like to know I am right and what others are thinking:
I have used Cyrus 2.2 for a long time before I switched to Dovecot; in Cyrus 2.2, I did not have the problem. So I am wondering how the problem has been solved there.
Obviously, users can create new folders in public namespace (if the ACL allows) and their private mailboxes; furthermore, they can delete folders from public namespace and private mailboxes (if the ACL allows) (I am talking of a pure delete here, not of moving the folder to Trash).
Thus, we could emulate moving folders between any namespaces (for example, from public namespace to private Trash) by first copying them to the new location and then deleting them from the original location.
Let's assume we have a folder in a public namespace and are allowed to do everything with it. Then, we could delete this folder (this time, I am meaning moving it to private Trash) by doing the following steps manually:
a) Create a new folder with the same name in Trash (which may be in the private mailbox); recursively repeat this with all subfolders.
b) Move all messages from the folder in the public namespace to the folder in Trash; recursively repeat this with all subfolders (remark: moving messages does not seem to cause problems, as opposed to moving folders).
c) Recursively delete the folder with all subfolders (which are completely empty now) from the public namespace (of course, this again is a "real" delete).
I have tested this procedure with multiple IMAP clients against Dovecot, and there were no problems. Automation of the procedure would not be expected to lead to problems, would it?.
- From a user's point of view, it is not important how messages get deleted or moved behind the scenes. A users which wants his stuff moved doesn't care if moving is accomplished by a rename or by another mechanism. In fact, he doesn't even know about renaming. He just wants the stuff which was in some folder to show up in some other folder.
It seems that the problems come from the fact that deletion is done by renaming. I am not in IMAP but suppose that there is some rename command which is used by email clients in such cases. Maybe there is also a delete command which dovecot is "emulating" by doing a rename - I really don't know...
So why not implement the rename / delete command in the way that I have described above? I know that a rename probably (for example on a file system or Dovecot's backends) is by far much more efficient than the copy-move-delete mechanism, but we could do the following:
If the renaming is problematic due to reasons Timo has mentioned in his first reply, then emulate the renaming by the copy-move-delete mechanism; if renaming would be no problem, then use the normal method (the method which is already implemented).
The inefficiency of the copy-move-delete mechanism would be no argument against using it: the alternative would be having the user to do these steps by hand, for every single subfolder (there may be thousands of them...). This is just not acceptable.
One could argue that this functionality should be in the client, and indeed, I suspect that some clients are working that way in some situations. Nevertheless, I think such clients are bad, by using inefficient operations where it would not be necessary and putting the servers under load.
IMHO, such functionality belongs into the server side since only the server can reliably decide if there will be problems before trying the operation.
I am absolutely sure users would prefer waiting for inefficient copy-move-delete operations instead of not being able to move their stuff at all.
Perhaps implementing such a behaviour would not be too difficult?
At the end, I would like to stress that this is not just an academic question. The need to move folders, especially between public and private namespaces, is very common in small and large companies, and although my original post was about deleting folders, the problem in fact is a real big one (although some clients seem to circumvent it by wasting server resources).
I am curiously looking forward to any comments.
Regards,
Peter
On 5/25/2011 5:12 AM, Binarus wrote:
It seems that the problems come from the fact that deletion is done by renaming. I am not in IMAP but suppose that there is some rename command which is used by email clients in such cases. Maybe there is also a delete command which dovecot is "emulating" by doing a rename - I really don't know...
Just a little correction here: Deletion is NOT done by renaming. Deletion of folders is done with a DELETE command which gets rid of it entirely. What your mail clients are doing is renaming the folder, which is done with a RENAME command.
For example, if "SomeFolder" is deleted then your mail client is doing: RENAME "SomeFolder" "Trash.SomeFolder" which effectively just moves the folder as a subfolder of Trash.
So, mail clients are asking to rename the folder (not delete it) in the case you are concerned about.
Timo, you said
Talking only about ACLs I guess it would be ok to allow renaming from public namespace to a private namespace. Renaming one of your own private mailboxes to public namespace is a bit trickier, because the default ACL is "no permissions". So when renaming, a new explicit ACL should be added to the user who did the renaming, because otherwise user could no longer access that mailbox at all. Still doable I guess.
I don't think any ACL should be added automatically in this case. If they set their trash folder to be in a public/shared namespace for which they have only CREATE permissions, then I think that's their fault and the admins problem... not Dovecot's.
On 26.05.2011 08:38, Willie Gillespie wrote:
On 5/25/2011 5:12 AM, Binarus wrote:
It seems that the problems come from the fact that deletion is done by renaming. I am not in IMAP but suppose that there is some rename command which is used by email clients in such cases. Maybe there is also a delete command which dovecot is "emulating" by doing a rename - I really don't know...
Just a little correction here: Deletion is NOT done by renaming. Deletion of folders is done with a DELETE command which gets rid of it entirely. What your mail clients are doing is renaming the folder, which is done with a RENAME command.
For example, if "SomeFolder" is deleted then your mail client is doing: RENAME "SomeFolder" "Trash.SomeFolder" which effectively just moves the folder as a subfolder of Trash.
I suspected something like that, and you can see it from the fact that some IMAP clients indeed are able to do the moving (obviously by internally emulating it by the copy-move-delete method).
That was the reason why I was proposing the server-side copy-move-delete implementation for such cases. I think it is the correct behaviour for clients to use renaming instead of more inefficient operations, but I also think that the server should try to obey the user and should emulate renaming by the copy-move-delete method if a "pure" renaming is not possible for some reason.
Obviously, Dovecot incorporates code for creating new folders in any namespace, for moving messages regardless of namespaces and for deleting (empty?) folders in any namespace (we can do all these actions manually provided the ACLs allow). Therefore, emulating renaming by copy-move-delete in problematic cases might be possible without pain.
I know that it is not a completely "clean" solution; on the other hand, it would be reasonable and immediately understandable by users. The purists among us might be happy if there was a configuration directive to enable or disable such behaviour.
What do you think about that?
Timo, you said
Talking only about ACLs I guess it would be ok to allow renaming from public namespace to a private namespace. Renaming one of your own private mailboxes to public namespace is a bit trickier, because the default ACL is "no permissions". So when renaming, a new explicit ACL should be added to the user who did the renaming, because otherwise user could no longer access that mailbox at all. Still doable I guess.
I don't think any ACL should be added automatically in this case. If they set their trash folder to be in a public/shared namespace for which they have only CREATE permissions, then I think that's their fault and the admins problem... not Dovecot's.
I agree with Willie. Once again, look at the analogy: what does a user expect to happen when he tries to move a folder from his private space to public space? He expects
- a new folder (with subfolder structure) to be created in public, every folder having the same name as it's counterpart in private
- all messages to be moved from the folder structure in private to the folder structure in public
- the folder structure in private to be deleted
These steps could be done by hand, and thus, the behaviour, ACL handling and error messages should exactly be the same as when doing the steps by hand. For example, when a user creates a folder in public by hand, he eventually will not be able to access that folder, depending of the ACLs of the parent folder. This is normal and expected, so there is no reason to handle it in special way when moving (renaming?) folders.
By the way, if I got it right, the same problem might appear if the user was copying folders within the same public namespace. What if a user copies a folder (in public) to another parent folder (also in public) where the user only has create rights? I don't know for sure, but I think the ACL won't be copied in this case, so the user might not have access to the new folder. But I may be wrong here, and I am unsure what would happen if the folder would be moved instead of copied; I think if moving is done by renaming, the ACL would be moved as well, so there should not be any problem.
We also could look at it the other way: Logically, from my (naive) point of view, any folder in a user's private namespace is connected with a standard ACL; however, this ACL can't be altered by users (or the admin), has only one entry (respective user has full access) and is created / kept automatically by the server. In this model, renaming from private to public would be the same as renaming from public to public and thus would solve the problems.
I don't know how difficult it would be to make Dovecot look at things that way. Maybe my point of view is completely wrong, but then I hope somebody will tell me about it...
Personally, I could live very well with both solutions (emulating moving by copy-move-delete in problematic cases or handling private namespace just as public namespace with special ACLs that can't be altered). I don't know which is easier, more logic, and if any of them completely respects RFCs.
But I know for sure that being unable to move folders between any namespaces is a big problem for users and even a bigger one for admins (guess what a user does if he can't move this one, super-important folder with hundreds of subfolders - this usually ends with the admin trying to move the whole thing on file-system level, of course during work time where Dovecot can't be shut down, or the admin desperately trying to find and configure an IMAP client which emulates moving internally by copy-move-delete).
Regards,
Peter
participants (3)
-
Binarus
-
Timo Sirainen
-
Willie Gillespie