Mapping of INBOX.INBOX breaks imapc_list_prefix=INBOX?
We're running a huge imapc-Migration from an old Courier-IMAP to Dovecot.
Courier always uses "INBOX."-prefix for his folders, so we set
namespace inbox { [...] separator = . prefix = INBOX. [...] }
on our new Dovecot 2.2.20 system.
And also we added:
imapc_list_prefix=INBOX
to avoid the dublication of the INBOX prefix.
At the end, doveadm/dsync/imapc see "INBOX.Virus", adds his Prefix, has "INBOX.INBOX.Virus", deletes his imapc_list_prefix and the end we have "INBOX.Virus" again which is identical to the source folder from Courier.
Works perfect BUT:
It looks like there is a "hidden feature" in Dovecot that always maps "INBOX.INBOX" back to "INBOX". For me that looks like a workaround for broken mailclients to fix some of the common problems after changing the namespace prefix.
I never heard about that "hidden feature", but even if I don't (!) have INBOX.INBOX:
root@imap01:~/MIGRATION # doveadm mailbox list -u test INBOX INBOX.Virus INBOX.Spam INBOX.Trash
I'm always able to access INBOX.INBOX which is identical to INBOX:
root@imap01:~/MIGRATION # doveadm fetch -u test uid mailbox INBOX.INBOX uid: 1
Looks like there is such a magic-mapping and looks like this "hard coded mapping" (?) breaks the imapc_list_prefix-magic.
If the user has on the SOURCE-server (Courier):
Source-Server: INBOX (5 messages) INBOX.Inbox (3 messages)
and I do migrate that user using doveadm/dsync/imapc, this user has on the DESTINATION-server (Dovecot)
Destination-server: INBOX (5 messages) INBOX.Inbox (5 messages from "real INBOX")
and his 3 messages from INBOX.Inbox are lost.
I can NOT find any errors in the debug/error-output of doveadm/dsync/imapc. From the view of doveadm everything worked perfect as you can see in the attached logfile.
Or am I completly wrong?!
Peer
-- Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin
http://www.heinlein-support.de
Tel: 030 / 405051-42 Fax: 030 / 405051-19
Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin
On 06 Jan 2016, at 11:47, Peer Heinlein <p.heinlein@heinlein-support.de> wrote:
We're running a huge imapc-Migration from an old Courier-IMAP to Dovecot.
Courier always uses "INBOX."-prefix for his folders, so we set
namespace inbox { [...] separator = . prefix = INBOX. [...] }
on our new Dovecot 2.2.20 system.
And also we added:
imapc_list_prefix=INBOX
to avoid the dublication of the INBOX prefix.
At the end, doveadm/dsync/imapc see "INBOX.Virus", adds his Prefix, has "INBOX.INBOX.Virus", deletes his imapc_list_prefix and the end we have "INBOX.Virus" again which is identical to the source folder from Courier.
Works perfect BUT:
It looks like there is a "hidden feature" in Dovecot that always maps "INBOX.INBOX" back to "INBOX". For me that looks like a workaround for broken mailclients to fix some of the common problems after changing the namespace prefix.
I never heard about that "hidden feature", but even if I don't (!) have INBOX.INBOX:
root@imap01:~/MIGRATION # doveadm mailbox list -u test INBOX INBOX.Virus INBOX.Spam INBOX.Trash
I'm always able to access INBOX.INBOX which is identical to INBOX:
root@imap01:~/MIGRATION # doveadm fetch -u test uid mailbox INBOX.INBOX uid: 1
Looks like there is such a magic-mapping and looks like this "hard coded mapping" (?) breaks the imapc_list_prefix-magic.
If the user has on the SOURCE-server (Courier):
Source-Server: INBOX (5 messages) INBOX.Inbox (3 messages)
and I do migrate that user using doveadm/dsync/imapc, this user has on the DESTINATION-server (Dovecot)
Destination-server: INBOX (5 messages) INBOX.Inbox (5 messages from "real INBOX")
and his 3 messages from INBOX.Inbox are lost.
I can NOT find any errors in the debug/error-output of doveadm/dsync/imapc. From the view of doveadm everything worked perfect as you can see in the attached logfile.
Or am I completly wrong?!
It's not really a "hidden feature", but more likely a consequence of the INBOX. prefix being added/removed from folder names. I'd need to look into it more carefully to see if it's possible to fix easily or not, but I have a feeling it may not be easy. INBOX.INBOX has caused troubles many times earlier.. One solution that should work is that for dsync runs you'd simply remove the INBOX. prefix from both the namespace and the imapc_list_prefix.
On 06 Jan 2016, at 15:35, Timo Sirainen <tss@iki.fi> wrote:
On 06 Jan 2016, at 11:47, Peer Heinlein <p.heinlein@heinlein-support.de> wrote:
We're running a huge imapc-Migration from an old Courier-IMAP to Dovecot.
Courier always uses "INBOX."-prefix for his folders, so we set
namespace inbox { [...] separator = . prefix = INBOX. [...] }
on our new Dovecot 2.2.20 system.
And also we added:
imapc_list_prefix=INBOX
to avoid the dublication of the INBOX prefix.
At the end, doveadm/dsync/imapc see "INBOX.Virus", adds his Prefix, has "INBOX.INBOX.Virus", deletes his imapc_list_prefix and the end we have "INBOX.Virus" again which is identical to the source folder from Courier.
Works perfect BUT:
It looks like there is a "hidden feature" in Dovecot that always maps "INBOX.INBOX" back to "INBOX". For me that looks like a workaround for broken mailclients to fix some of the common problems after changing the namespace prefix.
I never heard about that "hidden feature", but even if I don't (!) have INBOX.INBOX:
root@imap01:~/MIGRATION # doveadm mailbox list -u test INBOX INBOX.Virus INBOX.Spam INBOX.Trash
I'm always able to access INBOX.INBOX which is identical to INBOX:
root@imap01:~/MIGRATION # doveadm fetch -u test uid mailbox INBOX.INBOX uid: 1
Looks like there is such a magic-mapping and looks like this "hard coded mapping" (?) breaks the imapc_list_prefix-magic.
If the user has on the SOURCE-server (Courier):
Source-Server: INBOX (5 messages) INBOX.Inbox (3 messages)
and I do migrate that user using doveadm/dsync/imapc, this user has on the DESTINATION-server (Dovecot)
Destination-server: INBOX (5 messages) INBOX.Inbox (5 messages from "real INBOX")
and his 3 messages from INBOX.Inbox are lost.
I can NOT find any errors in the debug/error-output of doveadm/dsync/imapc. From the view of doveadm everything worked perfect as you can see in the attached logfile.
Or am I completly wrong?!
It's not really a "hidden feature", but more likely a consequence of the INBOX. prefix being added/removed from folder names. I'd need to look into it more carefully to see if it's possible to fix easily or not, but I have a feeling it may not be easy. INBOX.INBOX has caused troubles many times earlier.. One solution that should work is that for dsync runs you'd simply remove the INBOX. prefix from both the namespace and the imapc_list_prefix.
Or actually that would just add INBOX. prefix to everything. Maybe removing either namespace prefix or imapc_list_prefix would work?
Am 06.01.2016 um 22:03 schrieb Timo Sirainen:
Hi Timo,
thanks for your fast reply.
It's not really a "hidden feature", but more likely a consequence of the INBOX. prefix being added/removed from folder names. I'd need to look into it more carefully to see if it's possible to fix easily or not, but I have a feeling it may not be easy. INBOX.INBOX has caused troubles many times earlier..
Yes, I know that problems and I had them already many times (and learned how to avoid them). That's why I had the idea that there could be such a mapping and totally understand why it could be interesting to have this.
But doveadm isn't allowed to be confused by that mapping.
One solution that should work is that for dsync runs you'd simply remove the INBOX. prefix from both the namespace and the imapc_list_prefix.
I already tried this before, but I wouldn't be happy with that. We have two months migration time with a live migration and I do not want to offer a non-prefixed namespace to migrated users. I wouldn't be able to switch back to a prefixed namespace without all that huge problems caused by namespace changes.
AND: After adding the Prefix again I would have to fix subscriptions and I do have the risk that several hundred thousand users would try to refresh their local IMAP-cache simultaneously and would reload 25 TB data simultaneously. I can not accept that risk. :-)
Or actually that would just add INBOX. prefix to everything. Maybe removing either namespace prefix or imapc_list_prefix would work?
No.
But, anyway: Once you have approved that there IS and should be that kind of mapping and that THIS IS the reason of my problems, we know that we have to look (just) for users with folders named INBOX.INBOX and we'll find workarounds and scripts to handle them.
I think we easily rename that folders to INBOX.INBOX2 and everything will work fine.
There's no need for you to provide a fast fix just for us if the solution is not that easy. I just tried to confirm that bug AND to make sure that this is not just a stupid mistake by me.
Right at the moment we're running a find script to analyse how many users do have that double-INBOX-folder names. I don't expect that there will be too many users.
The only thing I still wonder is WHY the comparisation is done case insensitive. The source folder is named "INBOX.Inbox" and it's mixed with INBOX. As far as I experienced there's no need to expect anything else but INBOX after a namespace change. But maybe you have other experiences here. So there's no need for a discussion and no need to waste time.
Peer
-- Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin
http://www.heinlein-support.de
Tel: 030 / 405051-42 Fax: 030 / 405051-19
Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin
Am 06.01.2016 um 21:35 schrieb Timo Sirainen:
It's not really a "hidden feature", but more likely a consequence of the INBOX. prefix being added/removed from folder names.
I called this a "hidden feature" because for me that's a non-standard behavior without any options to switch that on or off.
Maybe a simple "double_inbox_mapping=yes|no" would solve the problem -- if this feature is off during migration there's no risk for that mapping mistake.
Peer
-- Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin
http://www.heinlein-support.de
Tel: 030 / 405051-42 Fax: 030 / 405051-19
Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin
participants (2)
-
Peer Heinlein
-
Timo Sirainen