dsync issues due to Prefix difference?
Hi everyone,
I'm hoping I am just missing something obvious - but I cannot seem to get a one-way migration dsync working between two Dovecot servers because the Prefix configuration is different.
I am doing a migration from our old Dovecot servers to some new cPanel based ones. Source: 2.2.15 Dest: 2.2.21
Here is the debug output for the sync:
doveadm(root): Debug: Loading modules from directory: /usr/lib/dovecot/modules/doveadm doveadm(root): Debug: Module loaded: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_sieve_plugin.so doveadm(testuser@example.ca): Debug: Effective uid=1322, gid=1322, home=/khstor3/khmail-mailboxes/l/li/example.ca/testuser doveadm(testuser@example.ca): Debug: Namespace inbox: type=private, prefix=, sep=, inbox=yes, hidden=no, list=yes, subscriptions=yes location=maildir:~/Maildir doveadm(testuser@example.ca): Debug: maildir++: root=/khstor3/khmail-mailboxes/l/li/example.ca/testuser/Maildir, index=, indexpvt=, control=, inbox=/khstor3/khmail-mailboxes/l/li/example.ca/testuser/Maildir, alt= doveadm(testuser@example.ca): Debug: Namespace : Using permissions from /khstor3/khmail-mailboxes/l/li/example.ca/testuser/Maildir: mode=0777 gid=default stdin: is not a tty dsync-local(testuser@example.ca): Debug: brain M: Local mailbox tree: INBOX guid=f055ac07d79cad508a7a000075e38adc uid_validity=1353555159 uid_next=2085 subs=no last_change=0 last_subs=0 dsync-local(testuser@example.ca): Debug: brain M: Local mailbox tree: Junk guid=703a8b1d51ceab56f91e0000667eee40 uid_validity=1391566942 uid_next=1 subs=yes last_change=0 last_subs=1454100049 dsync-local(testuser@example.ca): Debug: brain M: Local mailbox tree: Trash guid=9bee9f0f2aacb65039260000b171e44d uid_validity=1353555163 uid_next=25 subs=no last_change=0 last_subs=0 dsync-local(testuser@example.ca): Debug: brain M: Local mailbox tree: Drafts guid=4d03661058a0f1524f240000e9d81ad0 uid_validity=1391566936 uid_next=1 subs=no last_change=0 last_subs=0 dsync-local(testuser@example.ca): Debug: brain M: Local mailbox tree: Sent guid=99ee9f0f2aacb65039260000b171e44d uid_validity=1353555161 uid_next=4 subs=no last_change=0 last_subs=0 dsync-local(testuser@example.ca): Debug: brain M: Remote mailbox tree: INBOX guid=6460141741cfab5643470000afb0d608 uid_validity=1454100291 uid_next=2064 subs=no last_change=0 last_subs=0 dsync-local(testuser@example.ca): Debug: brain M: Remote mailbox tree: INBOX.Trash guid=6360141741cfab5643470000afb0d608 uid_validity=1454100290 uid_next=12 subs=yes last_change=0 last_subs=1454100289 dsync-local(testuser@example.ca): Debug: brain M: Remote mailbox tree: INBOX.Sent guid=6260141741cfab5643470000afb0d608 uid_validity=1454100289 uid_next=2 subs=yes last_change=0 last_subs=1454100289 dsync-remote(testuser@example.ca): Error: Couldn't find namespace for mailbox Junk
I believe the problem is because on the source we have no prefix set, but on the destination it has INBOX as the prefix. I need to use dsync because we have many POP3 clients so I need to ensure the UIDL's are copied correctly.
Here is the output from 'doveconf namespace':
Source: namespace inbox { disabled = no hidden = no ignore_on_failure = no inbox = yes list = yes location = mailbox Drafts { auto = subscribe driver = special_use = \Drafts } mailbox Junk { auto = subscribe driver = special_use = \Junk } mailbox Sent { auto = subscribe driver = special_use = \Sent } mailbox "Sent Messages" { auto = no driver = special_use = \Sent } mailbox Trash { auto = subscribe driver = special_use = \Trash } order = 0 prefix = separator = subscriptions = yes type = private }
Destination:
namespace { disabled = no hidden = no ignore_on_failure = no inbox = yes list = yes location = mailbox Archive { auto = create autoexpunge = 0 comment = driver = special_use = \Archive } mailbox Archives { auto = no autoexpunge = 0 comment = driver = special_use = \Archive } mailbox Drafts { auto = subscribe autoexpunge = 0 comment = driver = special_use = \Drafts } mailbox Junk { auto = create autoexpunge = 0 comment = driver = special_use = \Junk } mailbox Sent { auto = subscribe autoexpunge = 0 comment = driver = special_use = \Sent } mailbox "Sent Messages" { auto = no autoexpunge = 0 comment = driver = special_use = \Sent } mailbox Trash { auto = subscribe autoexpunge = 0 comment = driver = special_use = \Trash } order = 0 prefix = INBOX. separator = subscriptions = yes type = private }
If anyone could provide some assistance it would be greatly appreciated!
Thanks, Mark
On 29 Jan 2016, at 23:16, Mark Little <marklittle@koallo.com> wrote:
Hi everyone,
I'm hoping I am just missing something obvious - but I cannot seem to get a one-way migration dsync working between two Dovecot servers because the Prefix configuration is different.
I think a regular dsync -> dsync migration won't work then directly, because it assumes similar source and destination namespaces. One possibility would be that if you start dsync from the destination server, you simply point it to a configuration with a similar namespace config (via -c dovecot.conf) as the source. It should still write everything correctly so that afterwards you can run with the wanted the namespace prefix (i.e. prefix can be changed in dovecot.conf without having to modify any per-user files). Another possibility would be to do imapc sync and set imapc_prefix=INBOX. - see http://wiki2.dovecot.org/Migration/Dsync
Hi Timo,
Thank you very much for the information. I had seen the DSync page but I was confused as to how to configure it. I'd definitely love to be able to get the proper dsync migration working as I have some clients who use POP3 and others who use IMAP and I want to make this seamless for all if possible.
Could you please elaborate on how I specify the alternate dovecot.conf?
I've gone through the man pages and I don't see anything referencing -c
for doveadm? Or do you mean that I should be running the full dovecot
server process from the source server with an alternate configuration?
If I change the name space won't it fail to find the local folders in
their mailbox?
Cheers, Mark
On 2016-01-30 19:05, Timo Sirainen wrote:
On 29 Jan 2016, at 23:16, Mark Little <marklittle@koallo.com> wrote:
Hi everyone,
I'm hoping I am just missing something obvious - but I cannot seem to get a one-way migration dsync working between two Dovecot servers because the Prefix configuration is different.
I think a regular dsync -> dsync migration won't work then directly, because it assumes similar source and destination namespaces. One possibility would be that if you start dsync from the destination server, you simply point it to a configuration with a similar namespace config (via -c dovecot.conf) as the source. It should still write everything correctly so that afterwards you can run with the wanted the namespace prefix (i.e. prefix can be changed in dovecot.conf without having to modify any per-user files). Another possibility would be to do imapc sync and set imapc_prefix=INBOX. - see http://wiki2.dovecot.org/Migration/Dsync
On 01 Feb 2016, at 23:39, Mark Little <marklittle@koallo.com> wrote:
Hi Timo,
Thank you very much for the information. I had seen the DSync page but I was confused as to how to configure it. I'd definitely love to be able to get the proper dsync migration working as I have some clients who use POP3 and others who use IMAP and I want to make this seamless for all if possible.
Could you please elaborate on how I specify the alternate dovecot.conf? I've gone through the man pages and I don't see anything referencing -c for doveadm? Or do you mean that I should be running the full dovecot server process from the source server with an alternate configuration?
-c is a global option for all dovecot binaries, including doveadm. I guess all the global options should be included in all the man pages..
If I change the name space won't it fail to find the local folders in their mailbox?
I'm not sure what you mean here. It's possible to switch between namespaces on the fly without losing anything (besides clients seeing them differently).
Cheers, Mark
On 2016-01-30 19:05, Timo Sirainen wrote:
On 29 Jan 2016, at 23:16, Mark Little <marklittle@koallo.com> wrote: Hi everyone, I'm hoping I am just missing something obvious - but I cannot seem to get a one-way migration dsync working between two Dovecot servers because the Prefix configuration is different. I think a regular dsync -> dsync migration won't work then directly, because it assumes similar source and destination namespaces. One possibility would be that if you start dsync from the destination server, you simply point it to a configuration with a similar namespace config (via -c dovecot.conf) as the source. It should still write everything correctly so that afterwards you can run with the wanted the namespace prefix (i.e. prefix can be changed in dovecot.conf without having to modify any per-user files). Another possibility would be to do imapc sync and set imapc_prefix=INBOX. - see http://wiki2.dovecot.org/Migration/Dsync
On 2016-02-02 07:22, Timo Sirainen wrote:
On 01 Feb 2016, at 23:39, Mark Little <marklittle@koallo.com> wrote:
Hi Timo,
Thank you very much for the information. I had seen the DSync page but I was confused as to how to configure it. I'd definitely love to be able to get the proper dsync migration working as I have some clients who use POP3 and others who use IMAP and I want to make this seamless for all if possible.
Could you please elaborate on how I specify the alternate dovecot.conf? I've gone through the man pages and I don't see anything referencing -c for doveadm? Or do you mean that I should be running the full dovecot server process from the source server with an alternate configuration?
-c is a global option for all dovecot binaries, including doveadm. I guess all the global options should be included in all the man pages..
Ok great. I will try that out. So you are saying I could run the daemon on a source server specifying to use a different config with the namespace matching the other, and then when dsync connects it will see the namespaces as the same?
If I change the name space won't it fail to find the local folders in their mailbox?
I'm not sure what you mean here. It's possible to switch between namespaces on the fly without losing anything (besides clients seeing them differently).
Ok. I am just realizing that I am probably unable to have a transparent migration for IMAP, right? Seeing as my destination is using a different Prefix, that means some IMAP clients will not see their migrated folders by default unless they update their configuration.
I am mainly concerned about POP3 I guess, because if IMAP has to re-download it's not as big of a deal because everything is already marked as read.. but re-downloading POP3 doesn't work the same.
I tried following the dsync page but I cannot get it working. Do I just have to include the POP3 configuration examples, or the IMAP too? Do I need to use 'pop3c:' at the end of the line instead of imapc: (eg: 'doveadm -R -u user@domain pop3c:') to signify it needs to preserve the UIDL's?
Do I need to add the same settings to the source as well as destination? or just one side? I tried yesterday but then ended up breaking Dovecot on my destination server. After I added the pop3-migration namespace part to the namespace section, all users were failing with: 'Error: User initialization failed: namespace configuration error: list=yes namespace missing'
Sorry for all the questions but the I am unable to work it out from the documentation documentation.
If you know of some examples or some additional documentation it would be very appreciated.
Thanks, Mark
On 2016-01-30 19:05, Timo Sirainen wrote:
On 29 Jan 2016, at 23:16, Mark Little <marklittle@koallo.com> wrote: Hi everyone, I'm hoping I am just missing something obvious - but I cannot seem to get a one-way migration dsync working between two Dovecot servers because the Prefix configuration is different. I think a regular dsync -> dsync migration won't work then directly, because it assumes similar source and destination namespaces. One possibility would be that if you start dsync from the destination server, you simply point it to a configuration with a similar namespace config (via -c dovecot.conf) as the source. It should still write everything correctly so that afterwards you can run with the wanted the namespace prefix (i.e. prefix can be changed in dovecot.conf without having to modify any per-user files). Another possibility would be to do imapc sync and set imapc_prefix=INBOX. - see http://wiki2.dovecot.org/Migration/Dsync
Hi all,
Just wondering if anyone else has any experience with imapc: migration?
I am trying to get a POP3/IMAP migration going from another Dovecot
installation which has a different Prefix.
I am having trouble working out exactly what needs to be done in the configuration to get the migration working correctly. I have some POP3 users so I want to use the proper pop3 migration plugin but the documentation is a bit lacking.
If anyone has any experience doing this in the past: the end if I need the POP3 UIDLs to be preserved? (The documentation
- Do I need to include both imapc: AND pop3c: settings in the Dovecot config on the destination server?
- When initiating the migration, should I be using pop3c: or imapc: at
gives an example of "doveadm -R -u user@domain imapc:") Or if I have the migration plugin enabled, will it automatically do the POP3 stuff even if I specify "imapc:"?
Any help at all would be greatly appreciated!
Thank you.
On 2016-02-02 10:04, Mark Little wrote:
On 2016-02-02 07:22, Timo Sirainen wrote:
On 01 Feb 2016, at 23:39, Mark Little <marklittle@koallo.com> wrote:
Hi Timo,
Thank you very much for the information. I had seen the DSync page but I was confused as to how to configure it. I'd definitely love to be able to get the proper dsync migration working as I have some clients who use POP3 and others who use IMAP and I want to make this seamless for all if possible.
Could you please elaborate on how I specify the alternate dovecot.conf? I've gone through the man pages and I don't see anything referencing -c for doveadm? Or do you mean that I should be running the full dovecot server process from the source server with an alternate configuration?
-c is a global option for all dovecot binaries, including doveadm. I guess all the global options should be included in all the man pages..
Ok great. I will try that out. So you are saying I could run the daemon on a source server specifying to use a different config with the namespace matching the other, and then when dsync connects it will see the namespaces as the same?
If I change the name space won't it fail to find the local folders in their mailbox?
I'm not sure what you mean here. It's possible to switch between namespaces on the fly without losing anything (besides clients seeing them differently).
Ok. I am just realizing that I am probably unable to have a transparent migration for IMAP, right? Seeing as my destination is using a different Prefix, that means some IMAP clients will not see their migrated folders by default unless they update their configuration.
I am mainly concerned about POP3 I guess, because if IMAP has to re-download it's not as big of a deal because everything is already marked as read.. but re-downloading POP3 doesn't work the same.
I tried following the dsync page but I cannot get it working. Do I just have to include the POP3 configuration examples, or the IMAP too? Do I need to use 'pop3c:' at the end of the line instead of imapc: (eg: 'doveadm -R -u user@domain pop3c:') to signify it needs to preserve the UIDL's?
Do I need to add the same settings to the source as well as destination? or just one side? I tried yesterday but then ended up breaking Dovecot on my destination server. After I added the pop3-migration namespace part to the namespace section, all users were failing with: 'Error: User initialization failed: namespace configuration error: list=yes namespace missing'
Sorry for all the questions but the I am unable to work it out from the documentation documentation.
If you know of some examples or some additional documentation it would be very appreciated.
Thanks, Mark
On 2016-01-30 19:05, Timo Sirainen wrote:
On 29 Jan 2016, at 23:16, Mark Little <marklittle@koallo.com> wrote: Hi everyone, I'm hoping I am just missing something obvious - but I cannot seem to get a one-way migration dsync working between two Dovecot servers because the Prefix configuration is different. I think a regular dsync -> dsync migration won't work then directly, because it assumes similar source and destination namespaces. One possibility would be that if you start dsync from the destination server, you simply point it to a configuration with a similar namespace config (via -c dovecot.conf) as the source. It should still write everything correctly so that afterwards you can run with the wanted the namespace prefix (i.e. prefix can be changed in dovecot.conf without having to modify any per-user files). Another possibility would be to do imapc sync and set imapc_prefix=INBOX. - see http://wiki2.dovecot.org/Migration/Dsync
On 02 Feb 2016, at 17:04, Mark Little <marklittle@koallo.com> wrote:
On 2016-02-02 07:22, Timo Sirainen wrote:
On 01 Feb 2016, at 23:39, Mark Little <marklittle@koallo.com> wrote: Hi Timo, Thank you very much for the information. I had seen the DSync page but I was confused as to how to configure it. I'd definitely love to be able to get the proper dsync migration working as I have some clients who use POP3 and others who use IMAP and I want to make this seamless for all if possible. Could you please elaborate on how I specify the alternate dovecot.conf? I've gone through the man pages and I don't see anything referencing -c for doveadm? Or do you mean that I should be running the full dovecot server process from the source server with an alternate configuration? -c is a global option for all dovecot binaries, including doveadm. I guess all the global options should be included in all the man pages..
Ok great. I will try that out. So you are saying I could run the daemon on a source server specifying to use a different config with the namespace matching the other, and then when dsync connects it will see the namespaces as the same?
I think my idea was that you keep the source server configuration as it is. The in destination server you have a dovecot-dsync.conf which is same as the regular dovecot.conf except you change the namespace prefix. Then run doveadm -c dovecot-dsync.conf sync ..
If I change the name space won't it fail to find the local folders in their mailbox? I'm not sure what you mean here. It's possible to switch between namespaces on the fly without losing anything (besides clients seeing them differently).
Ok. I am just realizing that I am probably unable to have a transparent migration for IMAP, right? Seeing as my destination is using a different Prefix, that means some IMAP clients will not see their migrated folders by default unless they update their configuration.
Right.
I am mainly concerned about POP3 I guess, because if IMAP has to re-download it's not as big of a deal because everything is already marked as read.. but re-downloading POP3 doesn't work the same.
I tried following the dsync page but I cannot get it working. Do I just have to include the POP3 configuration examples, or the IMAP too? Do I need to use 'pop3c:' at the end of the line instead of imapc: (eg: 'doveadm -R -u user@domain pop3c:') to signify it needs to preserve the UIDL's?
No. The dsync migration page should be correct.
Do I need to add the same settings to the source as well as destination? or just one side? I tried yesterday but then ended up breaking Dovecot on my destination server. After I added the pop3-migration namespace part to the namespace section, all users were failing with: 'Error: User initialization failed: namespace configuration error: list=yes namespace missing'
This looks as if the only namespace you had was the pop3-migration, and you're missing the inbox namespace? I added it now to the wiki example also.
participants (2)
-
Mark Little
-
Timo Sirainen