doveadm backup|sync works for every folder but INBOX
I'm trying to migrate an old Cyrus 2.5 server to Dovecot 2.3.19 using doveadm backup -R, which works for all folders but the INBOX itself, which always stays empty.
The Cyrus side uses altnamespace:no and unixhierarchysep:no, it's used as imapc: remote in doveadm backup -R with imapc_list_prefix=INBOX
Dovecot uses the following namespace to migrate into:
namespace inboxes { inbox = yes location = mailbox Sent { auto = subscribe special_use = \Sent } ### some more folders omitted ### prefix = INBOX/ separator = / subscriptions = no }
I use the following command line for the migration:
doveadm -o imapc_features='rfc822.size fetch-headers' -o mail_prefetch_count=20 -o namespace/subs/location=mbox:/var/dovecot/subs -o imapc_user='someuser' -o imapc_password='secret' -o imapc_host=<cyrus-ip> -o imapc_list_prefix=INBOX -D backup -f -n INBOX/ -R -u someuser@somedomain imapc:
The debug output contains the following lines:
Nov 02 10:16:01 dsync(someuser@somedomain): Debug: brain S: Namespace INBOX/ has location imapc: Nov 02 10:16:01 dsync(someuser@somedomain): Debug: brain M: Local mailbox tree: INBOX guid=d8d5a3122bce5f63ef9101001ebfc848 uid_validity=1667223083 uid_next=1 subs=no last_change=0 last_subs=0 Nov 02 10:16:01 dsync(someuser@somedomain): Debug: brain S: Local mailbox tree: INBOX guid=c92f64f79f0d1ed01e6d5b314f04886c uid_validity=1577952633 uid_next=32691 subs=no last_change=0 last_subs=0
Nov 02 10:16:28 dsync(someuser@somedomain): Warning: Mailbox changes caused a desync. You may want to run dsync again: Remote lost mailbox GUID c92f64f79f0d1ed01e6d5b314f04886c (maybe it was just deleted?)
The Cyrus server is solely used for the migration, no access from anywhere else!
doveadm mailbox status -u someuser@somedomain all INBOX reports:
INBOX messages=0 recent=0 uidnext=1 uidvalidity=1667223083 unseen=0 highestmodseq=1 vsize=0 guid=d8d5a3122bce5f63ef9101001ebfc848 firstsaved=never
As all other folders seem ok, I blamed the failure with the INBOX on the imapc_list_prefix and altnamespace:no and tried, adding the INBOX content with an other doveadm sync run:
doveadm -o imapc_features='rfc822.size fetch-headers' -o mail_prefetch_count=20 -o namespace/subs/location=mbox:/var/dovecot/subs -o imapc_user='someuser' -o imapc_password='secret' -o imapc_host=<cyrus-ip> -D sync -1 -f -m INBOX -n INBOX/ -R -u someuser@somedomain imapc:
That gives the following output:
Nov 02 10:55:45 dsync(someuser@somedomain): Debug: brain S: Namespace INBOX/ has location imapc: Nov 02 10:55:45 dsync(someuser@somedomain): Debug: brain M: Local mailbox tree: INBOX guid=d8d5a3122bce5f63ef9101001ebfc848 uid_validity=1667223083 uid_next=1 subs=no last_change=0 last_subs=0 Nov 02 10:55:45 dsync(someuser@somedomain): Debug: brain S: Local mailbox tree: INBOX guid=c92f64f79f0d1ed01e6d5b314f04886c uid_validity=1577952633 uid_next=32691 subs=no last_change=0 last_subs=0 Nov 02 10:55:45 dsync(someuser@somedomain): Debug: brain M: Remote mailbox tree: INBOX guid=c92f64f79f0d1ed01e6d5b314f04886c uid_validity=1577952633 uid_next=32691 subs=no last_change=0 last_subs=0 Nov 02 10:55:45 dsync(someuser@somedomain): Debug: brain S: Remote mailbox tree: INBOX guid=d8d5a3122bce5f63ef9101001ebfc848 uid_validity=1667223083 uid_next=1 subs=no last_change=0 last_subs=0 Nov 02 10:55:45 dsync(someuser@somedomain): Debug: brain M: Mailbox INBOX: local=d8d5a3122bce5f63ef9101001ebfc848/0/1, remote=c92f64f79f0d1ed01e6d5b314f04886c/0/1: GUIDs conflict - will be merged later Nov 02 10:55:45 dsync(someuser@somedomain): Debug: Mailbox INBOX: Mailbox opened Nov 02 10:55:45 dsync(someuser@somedomain): Debug: brain M: Changing mailbox INBOX GUID d8d5a3122bce5f63ef9101001ebfc848 -> c92f64f79f0d1ed01e6d5b314f04886c Nov 02 10:55:45 dsync(someuser@somedomain): Debug: dict(file): Waiting for dict to finish pending operations Nov 02 10:55:45 dsync(someuser@somedomain): Debug: brain S: Remote mailbox tree deletion: guid=c8882c0029ce5f63a9aa0000ca9feb0b type=mailbox timestamp=1667223081 name= local update=not found Nov 02 10:55:45 dsync(someuser@somedomain): Debug: brain S: Mailbox INBOX: local=c92f64f79f0d1ed01e6d5b314f04886c/0/1, remote=d8d5a3122bce5f63ef9101001ebfc848/0/1: GUIDs conflict - will be merged later Nov 02 10:55:45 dsync(someuser@somedomain): Debug: Mailbox INBOX: Mailbox opened Nov 02 10:55:45 dsync(someuser@somedomain): Debug: brain S: Ignore nonexistent mailbox GUID c92f64f79f0d1ed01e6d5b314f04886c with -1 sync Nov 02 10:55:45 dsync(someuser@somedomain): Debug: brain S: We don't have mailbox c92f64f79f0d1ed01e6d5b314f04886c Nov 02 10:55:45 dsync(someuser@somedomain): Debug: brain M: Ignoring missing remote box GUID c92f64f79f0d1ed01e6d5b314f04886c Nov 02 10:55:45 dsync(someuser@somedomain): Debug: dict(file): Waiting for dict to finish pending operations Nov 02 10:55:45 dsync(someuser@somedomain): Debug: dict(file): dict destroyed Nov 02 10:55:45 dsync(someuser@somedomain): Debug: dict(file): Waiting for dict to finish pending operations Nov 02 10:55:45 dsync(someuser@somedomain): Debug: dict(file): dict destroyed Nov 02 10:55:45 dsync(someuser@somedomain): Debug: imapc(10.44.88.3:143): Disconnected Nov 02 10:55:45 dsync(someuser@somedomain): Debug: User session is finished Nov 02 10:55:45 dsync(someuser@somedomain): Debug: dict(file): dict destroyed Nov 02 10:55:45 dsync(someuser@somedomain): Debug: dict(file): Waiting for dict to finish pending operations Nov 02 10:55:45 dsync(someuser@somedomain): Debug: dict(file): dict destroyed Nov 02 10:55:45 doveadm(40708): Debug: auth-master: conn unix:/run/dovecot/auth-userdb (pid=1,uid=0): Disconnected: Connection closed (fd=9)
doveadm mailbox status -u someuser@somedomain all INBOX INBOX messages=0 recent=0 uidnext=1 uidvalidity=1667223083 unseen=0 highestmodseq=1 vsize=0 guid=c92f64f79f0d1ed01e6d5b314f04886c firstsaved=never
The interesting part is that INBOX somehow seems to have two different GUIDs, depending on how I sync it (compare the two doveadm mailbox status outputs).
In a tcpdump of the IMAP trafic I can see a STATUS "INBOX" (UIDNEXT UIDVALIDITY) but no EXAMINE "INBOX" or something else I would expect from the sync.
I can provide the full output of doveadm backup/sync and a pcap of the IMAP trafic, but not to the public list.
Any idea what's wrong with the sync of the INBOX or other suggestions / imapc parameters to use?
Ralf
Ralf Becker EGroupware GmbH [www.egroupware.org] Handelsregister HRB Kaiserslautern 3587 Geschäftsführer Birgit und Ralf Becker Leibnizstr. 17, 67663 Kaiserslautern, Germany Telefon +49 631 31657-0
On 03/11/2022 09:46 EET Ralf Becker rb@egroupware.org wrote:
I'm trying to migrate an old Cyrus 2.5 server to Dovecot 2.3.19 using doveadm backup -R, which works for all folders but the INBOX itself, which always stays empty.
The Cyrus side uses altnamespace:no and unixhierarchysep:no, it's used as imapc: remote in doveadm backup -R with imapc_list_prefix=INBOX
Dovecot uses the following namespace to migrate into:
namespace inboxes { inbox = yes location = mailbox Sent { auto = subscribe special_use = \Sent } ### some more folders omitted ### prefix = INBOX/ separator = / subscriptions = no }
Hi!
When syncing mailboxes from other server, you should use migration config file, which has **no** auto=subscribe or auto=create folders, as these can mess up with synchronization.
Please see https://doc.dovecot.org/admin_manual/migrating_mailboxes/ for more details.
Aki
Hi Aki,
Am 03.11.22 um 08:50 schrieb Aki Tuomi:
On 03/11/2022 09:46 EET Ralf Becker rb@egroupware.org wrote:
I'm trying to migrate an old Cyrus 2.5 server to Dovecot 2.3.19 using doveadm backup -R, which works for all folders but the INBOX itself, which always stays empty.
The Cyrus side uses altnamespace:no and unixhierarchysep:no, it's used as imapc: remote in doveadm backup -R with imapc_list_prefix=INBOX
Dovecot uses the following namespace to migrate into:
namespace inboxes { inbox = yes location = mailbox Sent { auto = subscribe special_use = \Sent } ### some more folders omitted ### prefix = INBOX/ separator = / subscriptions = no } Hi!
When syncing mailboxes from other server, you should use migration config file, which has **no** auto=subscribe or auto=create folders, as these can mess up with synchronization.
Please see https://doc.dovecot.org/admin_manual/migrating_mailboxes/ for more details.
Does a migration config file specified with doveadm -c <file> add to and overwrite the existing Dovecot configuration for the time the command runs, like the -o options, or do I need to start a separate server with a full configuration to e.g. have my authentication and mailbox location available?
Ralf
-- Ralf Becker EGroupware GmbH [www.egroupware.org] Handelsregister HRB Kaiserslautern 3587 Geschäftsführer Birgit und Ralf Becker Leibnizstr. 17, 67663 Kaiserslautern, Germany Telefon +49 631 31657-0
On 03/11/2022 10:09 EET Ralf Becker rb@egroupware.org wrote:
Hi Aki,
Am 03.11.22 um 08:50 schrieb Aki Tuomi:
On 03/11/2022 09:46 EET Ralf Becker rb@egroupware.org wrote:
I'm trying to migrate an old Cyrus 2.5 server to Dovecot 2.3.19 using doveadm backup -R, which works for all folders but the INBOX itself, which always stays empty.
The Cyrus side uses altnamespace:no and unixhierarchysep:no, it's used as imapc: remote in doveadm backup -R with imapc_list_prefix=INBOX
Dovecot uses the following namespace to migrate into:
namespace inboxes { inbox = yes location = mailbox Sent { auto = subscribe special_use = \Sent } ### some more folders omitted ### prefix = INBOX/ separator = / subscriptions = no } Hi!
When syncing mailboxes from other server, you should use migration config file, which has **no** auto=subscribe or auto=create folders, as these can mess up with synchronization.
Please see https://doc.dovecot.org/admin_manual/migrating_mailboxes/ for more details.
Does a migration config file specified with doveadm -c <file> add to and overwrite the existing Dovecot configuration for the time the command runs, like the -o options, or do I need to start a separate server with a full configuration to e.g. have my authentication and mailbox location available?
Ralf
It does not add/replace/overwrite configuration, you provide a fresh config file which is used *instead of* the default dovecot.conf.
You don't need to run a separate instance necessarely, although in some larger migrations this has been used as well.
Aki
Hi Aki,
Am 03.11.22 um 09:12 schrieb Aki Tuomi:
On 03/11/2022 10:09 EET Ralf Becker rb@egroupware.org wrote:
Hi Aki,
Am 03.11.22 um 08:50 schrieb Aki Tuomi:
On 03/11/2022 09:46 EET Ralf Becker rb@egroupware.org wrote:
I'm trying to migrate an old Cyrus 2.5 server to Dovecot 2.3.19 using doveadm backup -R, which works for all folders but the INBOX itself, which always stays empty.
The Cyrus side uses altnamespace:no and unixhierarchysep:no, it's used as imapc: remote in doveadm backup -R with imapc_list_prefix=INBOX
Dovecot uses the following namespace to migrate into:
namespace inboxes { inbox = yes location = mailbox Sent { auto = subscribe special_use = \Sent } ### some more folders omitted ### prefix = INBOX/ separator = / subscriptions = no } Hi!
When syncing mailboxes from other server, you should use migration config file, which has **no** auto=subscribe or auto=create folders, as these can mess up with synchronization.
Please see https://doc.dovecot.org/admin_manual/migrating_mailboxes/ for more details. Does a migration config file specified with doveadm -c <file> add to and overwrite the existing Dovecot configuration for the time the command runs, like the -o options, or do I need to start a separate server with a full configuration to e.g. have my authentication and mailbox location available?
Ralf
It does not add/replace/overwrite configuration, you provide a fresh config file which is used *instead of* the default dovecot.conf.
You don't need to run a separate instance necessarely, although in some larger migrations this has been used as well.
I created now a separate instance with a modified configuration file with no auto=subscribe (or create), no replication and an empty storage. doveadm config -n is attached.
Unfortunately the result is identical to my previous tries:
doveadm -o namespace/subs/location=mbox:/var/dovecot/subs -o imapc_user='someuser' -o imapc_password='secret' -D backup -n INBOX/ -R -u someuser@somedomain imapc: 2>&1 | tee /tmp/doveadm-backup.log
Nov 03 09:06:35 dsync(someuser@somedomain): Warning: Mailbox changes caused a desync. You may want to run dsync again: Remote lost mailbox GUID c92f64f79f0d1ed01e6d5b314f04886c (maybe it was just deleted?)
doveadm mailbox status -u someuser@somedomain all INBOX INBOX messages=0 recent=0 uidnext=1 uidvalidity=1577952633 unseen=0 highestmodseq=1 vsize=0 guid=c92f64f79f0d1ed01e6d5b314f04886c firstsaved=never
Any ideas what else to try or how to debug that further?
I can send you the full log to your personal address, if that helps ...
Ralf
-- Ralf Becker EGroupware GmbH [www.egroupware.org] Handelsregister HRB Kaiserslautern 3587 Geschäftsführer Birgit und Ralf Becker Leibnizstr. 17, 67663 Kaiserslautern, Germany Telefon +49 631 31657-0
On 03/11/2022 11:27 EET Ralf Becker rb@egroupware.org wrote:
Hi Aki,
Am 03.11.22 um 09:12 schrieb Aki Tuomi:
On 03/11/2022 10:09 EET Ralf Becker rb@egroupware.org wrote:
Hi Aki,
Am 03.11.22 um 08:50 schrieb Aki Tuomi:
On 03/11/2022 09:46 EET Ralf Becker rb@egroupware.org wrote:
I'm trying to migrate an old Cyrus 2.5 server to Dovecot 2.3.19 using doveadm backup -R, which works for all folders but the INBOX itself, which always stays empty.
The Cyrus side uses altnamespace:no and unixhierarchysep:no, it's used as imapc: remote in doveadm backup -R with imapc_list_prefix=INBOX
Dovecot uses the following namespace to migrate into:
namespace inboxes { inbox = yes location = mailbox Sent { auto = subscribe special_use = \Sent } ### some more folders omitted ### prefix = INBOX/ separator = / subscriptions = no } Hi!
When syncing mailboxes from other server, you should use migration config file, which has **no** auto=subscribe or auto=create folders, as these can mess up with synchronization.
Please see https://doc.dovecot.org/admin_manual/migrating_mailboxes/ for more details. Does a migration config file specified with doveadm -c <file> add to and overwrite the existing Dovecot configuration for the time the command runs, like the -o options, or do I need to start a separate server with a full configuration to e.g. have my authentication and mailbox location available?
Ralf
It does not add/replace/overwrite configuration, you provide a fresh config file which is used *instead of* the default dovecot.conf.
You don't need to run a separate instance necessarely, although in some larger migrations this has been used as well.
I created now a separate instance with a modified configuration file with no auto=subscribe (or create), no replication and an empty storage. doveadm config -n is attached.
Unfortunately the result is identical to my previous tries:
doveadm -o namespace/subs/location=mbox:/var/dovecot/subs -o imapc_user='someuser' -o imapc_password='secret' -D backup -n INBOX/ -R -u someuser@somedomain imapc: 2>&1 | tee /tmp/doveadm-backup.log
Nov 03 09:06:35 dsync(someuser@somedomain): Warning: Mailbox changes caused a desync. You may want to run dsync again: Remote lost mailbox GUID c92f64f79f0d1ed01e6d5b314f04886c (maybe it was just deleted?)
doveadm mailbox status -u someuser@somedomain all INBOX INBOX messages=0 recent=0 uidnext=1 uidvalidity=1577952633 unseen=0 highestmodseq=1 vsize=0 guid=c92f64f79f0d1ed01e6d5b314f04886c firstsaved=never
Any ideas what else to try or how to debug that further?
I can send you the full log to your personal address, if that helps ...
Ralf
--
You should rm -rf the target folder first. Can you attach doveadm -D backup logs? Check that it won't contain passwords.
Aki
Hi Aki,
Am 03.11.22 um 10:29 schrieb Aki Tuomi:
On 03/11/2022 11:27 EET Ralf Becker rb@egroupware.org wrote:
Hi Aki,
Am 03.11.22 um 09:12 schrieb Aki Tuomi:
On 03/11/2022 10:09 EET Ralf Becker rb@egroupware.org wrote:
Hi Aki,
Am 03.11.22 um 08:50 schrieb Aki Tuomi:
On 03/11/2022 09:46 EET Ralf Becker rb@egroupware.org wrote:
I'm trying to migrate an old Cyrus 2.5 server to Dovecot 2.3.19 using doveadm backup -R, which works for all folders but the INBOX itself, which always stays empty.
The Cyrus side uses altnamespace:no and unixhierarchysep:no, it's used as imapc: remote in doveadm backup -R with imapc_list_prefix=INBOX
Dovecot uses the following namespace to migrate into:
namespace inboxes { inbox = yes location = mailbox Sent { auto = subscribe special_use = \Sent } ### some more folders omitted ### prefix = INBOX/ separator = / subscriptions = no } Hi!
When syncing mailboxes from other server, you should use migration config file, which has **no** auto=subscribe or auto=create folders, as these can mess up with synchronization.
Please see https://doc.dovecot.org/admin_manual/migrating_mailboxes/ for more details. Does a migration config file specified with doveadm -c <file> add to and overwrite the existing Dovecot configuration for the time the command runs, like the -o options, or do I need to start a separate server with a full configuration to e.g. have my authentication and mailbox location available?
Ralf
It does not add/replace/overwrite configuration, you provide a fresh config file which is used *instead of* the default dovecot.conf.
You don't need to run a separate instance necessarely, although in some larger migrations this has been used as well. I created now a separate instance with a modified configuration file with no auto=subscribe (or create), no replication and an empty storage. doveadm config -n is attached.
Unfortunately the result is identical to my previous tries:
doveadm -o namespace/subs/location=mbox:/var/dovecot/subs -o imapc_user='someuser' -o imapc_password='secret' -D backup -n INBOX/ -R -u someuser@somedomain imapc: 2>&1 | tee /tmp/doveadm-backup.log
Nov 03 09:06:35 dsync(someuser@somedomain): Warning: Mailbox changes caused a desync. You may want to run dsync again: Remote lost mailbox GUID c92f64f79f0d1ed01e6d5b314f04886c (maybe it was just deleted?)
doveadm mailbox status -u someuser@somedomain all INBOX INBOX messages=0 recent=0 uidnext=1 uidvalidity=1577952633 unseen=0 highestmodseq=1 vsize=0 guid=c92f64f79f0d1ed01e6d5b314f04886c firstsaved=never
Any ideas what else to try or how to debug that further?
I can send you the full log to your personal address, if that helps ...
Ralf You should rm -rf the target folder first. Can you attach
doveadm -D backuplogs? Check that it won't contain passwords.
The mailbox directory did NOT exist before, therefore no need to rm -rf it.
I send the logs to your private address only, I feel not comfortable to post them on the list.
Ralf
-- Ralf Becker EGroupware GmbH [www.egroupware.org] Handelsregister HRB Kaiserslautern 3587 Geschäftsführer Birgit und Ralf Becker Leibnizstr. 17, 67663 Kaiserslautern, Germany Telefon +49 631 31657-0
On 03/11/2022 11:46 EET Ralf Becker rb@egroupware.org wrote:
Hi Aki,
Am 03.11.22 um 10:29 schrieb Aki Tuomi:
On 03/11/2022 11:27 EET Ralf Becker rb@egroupware.org wrote:
Hi Aki,
Am 03.11.22 um 09:12 schrieb Aki Tuomi:
On 03/11/2022 10:09 EET Ralf Becker rb@egroupware.org wrote:
Hi Aki,
Am 03.11.22 um 08:50 schrieb Aki Tuomi:
> On 03/11/2022 09:46 EET Ralf Becker rb@egroupware.org wrote: > >
> I'm trying to migrate an old Cyrus 2.5 server to Dovecot 2.3.19 using > doveadm backup -R, which works for all folders but the INBOX itself, > which always stays empty. > > The Cyrus side uses altnamespace:no and unixhierarchysep:no, it's used > as imapc: remote in doveadm backup -R with imapc_list_prefix=INBOX > > Dovecot uses the following namespace to migrate into: > > namespace inboxes { > inbox = yes > location = > mailbox Sent { > auto = subscribe > special_use = \Sent > } > ### some more folders omitted ### > prefix = INBOX/ > separator = / > subscriptions = no > } Hi!When syncing mailboxes from other server, you should use migration config file, which has **no** auto=subscribe or auto=create folders, as these can mess up with synchronization.
Please see https://doc.dovecot.org/admin_manual/migrating_mailboxes/ for more details. Does a migration config file specified with doveadm -c <file> add to and overwrite the existing Dovecot configuration for the time the command runs, like the -o options, or do I need to start a separate server with a full configuration to e.g. have my authentication and mailbox location available?
Ralf
It does not add/replace/overwrite configuration, you provide a fresh config file which is used *instead of* the default dovecot.conf.
You don't need to run a separate instance necessarely, although in some larger migrations this has been used as well. I created now a separate instance with a modified configuration file with no auto=subscribe (or create), no replication and an empty storage. doveadm config -n is attached.
Unfortunately the result is identical to my previous tries:
doveadm -o namespace/subs/location=mbox:/var/dovecot/subs -o imapc_user='someuser' -o imapc_password='secret' -D backup -n INBOX/ -R -u someuser@somedomain imapc: 2>&1 | tee /tmp/doveadm-backup.log
Nov 03 09:06:35 dsync(someuser@somedomain): Warning: Mailbox changes caused a desync. You may want to run dsync again: Remote lost mailbox GUID c92f64f79f0d1ed01e6d5b314f04886c (maybe it was just deleted?)
doveadm mailbox status -u someuser@somedomain all INBOX INBOX messages=0 recent=0 uidnext=1 uidvalidity=1577952633 unseen=0 highestmodseq=1 vsize=0 guid=c92f64f79f0d1ed01e6d5b314f04886c firstsaved=never
Any ideas what else to try or how to debug that further?
I can send you the full log to your personal address, if that helps ...
Ralf You should rm -rf the target folder first. Can you attach
doveadm -D backuplogs? Check that it won't contain passwords.The mailbox directory did NOT exist before, therefore no need to rm -rf it.
I send the logs to your private address only, I feel not comfortable to post them on the list.
Ralf
You did not delete the mailbox.
You are using **mbox** for subscription namespace, please don't.
Also
Nov 03 09:05:33 dsync(): Debug: brain M: Local mailbox tree: INBOX guid=c8adef115c84636335000000effb6190 uid_validity=1667466332 uid_next=1 subs=no last_change=0 last_subs=0 Nov 03 09:05:33 dsync(): Debug: brain S: Local mailbox tree: INBOX guid=c92f64f79f0d1ed01e6d5b314f04886c uid_validity=1577952633 uid_next=32746 subs=no last_change=0 last_subs=0
This clearly shows that you did not, in fact, rm -rf the user's mailboxes prir running backup. Can you please try again, and clean up the user from t target mailsystem before running backup again?
I would also strongly recommend not using ACL plugin while doing backup, unless you are backing up ACLs from source system.
Aki
hopefully this is NOT off topic
I too migrated from cyrus imap server, it was to say at least fun
you need to get the dovecot server up and running and testing with making a new account and get the folder structures working dependant on your required setups, as what seems to be indicated elsewhere in these posts (namespace, virtual users, dbpassword etc)
if you intend to run replication then both servers need to be setup and running fully before migrating. Don't put that off it just adds work afterwards.
at the end of the day
moving from cyrus to dovecot i used imapsync
which is a perl script to migrate, copy etc
see : https://imapsync.lamiral.info/
the one major note is the seperator, if you used '.' on cyrus you need to convert it to '-' or something else as those folders will not migrate into a default dovecot configuration and will get skipped.
ie : folders can not have a '.' in them on default dovecot.
Otherwise imapsync pretty much does the trick.
basically :
imapsync --host1 69.49.101.233 --user1 ap@unitedelevatorltd.com --password1 Password \
--host2 mail18.scom.ca --user2 ap@unitedelevatorltd.com --password2 Password \
--regextrans2 "s,\.,_,g"
note the regextrans2 expresion above handles the '.' to '-'
there is extensive documentation on the site i have just listed what i had to use to get an account to work and move over with out any data loss.
basically if you write a script to do the copy's on a users account per mailbox basis then things should migrate well.
ie : get user list / passwords from old cyrus
then create mailbox on new dovecot server (ie update password database etc)
then run imapsync like above.
repeat for every user you want to migrate.
I did not have any issues with the INBOX but after a month of testing found this was the best way to go.
Happy Thursday !!! Thanks - paul
Paul Kudla
Scom.ca Internet Services http://www.scom.ca 004-1009 Byron Street South Whitby, Ontario - Canada L1N 4S3
Toronto 416.642.7266 Main 1.866.411.7266 Fax 1.888.892.7266 Email paul@scom.ca
On 11/3/2022 5:54 AM, Aki Tuomi wrote:
On 03/11/2022 11:46 EET Ralf Becker rb@egroupware.org wrote:
Hi Aki,
Am 03.11.22 um 10:29 schrieb Aki Tuomi:
On 03/11/2022 11:27 EET Ralf Becker rb@egroupware.org wrote:
Hi Aki,
Am 03.11.22 um 09:12 schrieb Aki Tuomi:
On 03/11/2022 10:09 EET Ralf Becker rb@egroupware.org wrote:
Hi Aki,
Am 03.11.22 um 08:50 schrieb Aki Tuomi: >> On 03/11/2022 09:46 EET Ralf Becker rb@egroupware.org wrote: >> >>
>> I'm trying to migrate an old Cyrus 2.5 server to Dovecot 2.3.19 using >> doveadm backup -R, which works for all folders but the INBOX itself, >> which always stays empty. >> >> The Cyrus side uses altnamespace:no and unixhierarchysep:no, it's used >> as imapc: remote in doveadm backup -R with imapc_list_prefix=INBOX >> >> Dovecot uses the following namespace to migrate into: >> >> namespace inboxes { >> inbox = yes >> location = >> mailbox Sent { >> auto = subscribe >> special_use = \Sent >> } >> ### some more folders omitted ### >> prefix = INBOX/ >> separator = / >> subscriptions = no >> } > Hi! > > When syncing mailboxes from other server, you should use migration config file, which has **no** auto=subscribe or auto=create folders, as these can mess up with synchronization. > > Please see https://doc.dovecot.org/admin_manual/migrating_mailboxes/ for more details. Does a migration config file specified with doveadm -c <file> add to and overwrite the existing Dovecot configuration for the time the command runs, like the -o options, or do I need to start a separate server with a full configuration to e.g. have my authentication and mailbox location available?Ralf
It does not add/replace/overwrite configuration, you provide a fresh config file which is used *instead of* the default dovecot.conf.
You don't need to run a separate instance necessarely, although in some larger migrations this has been used as well. I created now a separate instance with a modified configuration file with no auto=subscribe (or create), no replication and an empty storage. doveadm config -n is attached.
Unfortunately the result is identical to my previous tries:
doveadm -o namespace/subs/location=mbox:/var/dovecot/subs -o imapc_user='someuser' -o imapc_password='secret' -D backup -n INBOX/ -R -u someuser@somedomain imapc: 2>&1 | tee /tmp/doveadm-backup.log
Nov 03 09:06:35 dsync(someuser@somedomain): Warning: Mailbox changes caused a desync. You may want to run dsync again: Remote lost mailbox GUID c92f64f79f0d1ed01e6d5b314f04886c (maybe it was just deleted?)
doveadm mailbox status -u someuser@somedomain all INBOX INBOX messages=0 recent=0 uidnext=1 uidvalidity=1577952633 unseen=0 highestmodseq=1 vsize=0 guid=c92f64f79f0d1ed01e6d5b314f04886c firstsaved=never
Any ideas what else to try or how to debug that further?
I can send you the full log to your personal address, if that helps ...
Ralf You should rm -rf the target folder first. Can you attach
doveadm -D backuplogs? Check that it won't contain passwords.The mailbox directory did NOT exist before, therefore no need to rm -rf it.
I send the logs to your private address only, I feel not comfortable to post them on the list.
Ralf
You did not delete the mailbox.
You are using **mbox** for subscription namespace, please don't.
Also
Nov 03 09:05:33 dsync(): Debug: brain M: Local mailbox tree: INBOX guid=c8adef115c84636335000000effb6190 uid_validity=1667466332 uid_next=1 subs=no last_change=0 last_subs=0 Nov 03 09:05:33 dsync(): Debug: brain S: Local mailbox tree: INBOX guid=c92f64f79f0d1ed01e6d5b314f04886c uid_validity=1577952633 uid_next=32746 subs=no last_change=0 last_subs=0
This clearly shows that you did not, in fact, rm -rf the user's mailboxes prir running backup. Can you please try again, and clean up the user from t target mailsystem before running backup again?
I would also strongly recommend not using ACL plugin while doing backup, unless you are backing up ACLs from source system.
Aki
Hi Aki,
Am 03.11.22 um 10:54 schrieb Aki Tuomi:
On 03/11/2022 11:46 EET Ralf Becker rb@egroupware.org wrote:
Hi Aki,
Am 03.11.22 um 10:29 schrieb Aki Tuomi:
On 03/11/2022 11:27 EET Ralf Becker rb@egroupware.org wrote:
Hi Aki,
Am 03.11.22 um 09:12 schrieb Aki Tuomi:
On 03/11/2022 10:09 EET Ralf Becker rb@egroupware.org wrote:
Hi Aki,
Am 03.11.22 um 08:50 schrieb Aki Tuomi: >> On 03/11/2022 09:46 EET Ralf Becker rb@egroupware.org wrote: >> >>
>> I'm trying to migrate an old Cyrus 2.5 server to Dovecot 2.3.19 using >> doveadm backup -R, which works for all folders but the INBOX itself, >> which always stays empty. >> >> The Cyrus side uses altnamespace:no and unixhierarchysep:no, it's used >> as imapc: remote in doveadm backup -R with imapc_list_prefix=INBOX >> >> Dovecot uses the following namespace to migrate into: >> >> namespace inboxes { >> inbox = yes >> location = >> mailbox Sent { >> auto = subscribe >> special_use = \Sent >> } >> ### some more folders omitted ### >> prefix = INBOX/ >> separator = / >> subscriptions = no >> } > Hi! > > When syncing mailboxes from other server, you should use migration config file, which has **no** auto=subscribe or auto=create folders, as these can mess up with synchronization. > > Please see https://doc.dovecot.org/admin_manual/migrating_mailboxes/ for more details. Does a migration config file specified with doveadm -c <file> add to and overwrite the existing Dovecot configuration for the time the command runs, like the -o options, or do I need to start a separate server with a full configuration to e.g. have my authentication and mailbox location available?Ralf
It does not add/replace/overwrite configuration, you provide a fresh config file which is used *instead of* the default dovecot.conf.
You don't need to run a separate instance necessarely, although in some larger migrations this has been used as well. I created now a separate instance with a modified configuration file with no auto=subscribe (or create), no replication and an empty storage. doveadm config -n is attached.
Unfortunately the result is identical to my previous tries:
doveadm -o namespace/subs/location=mbox:/var/dovecot/subs -o imapc_user='someuser' -o imapc_password='secret' -D backup -n INBOX/ -R -u someuser@somedomain imapc: 2>&1 | tee /tmp/doveadm-backup.log
Nov 03 09:06:35 dsync(someuser@somedomain): Warning: Mailbox changes caused a desync. You may want to run dsync again: Remote lost mailbox GUID c92f64f79f0d1ed01e6d5b314f04886c (maybe it was just deleted?)
doveadm mailbox status -u someuser@somedomain all INBOX INBOX messages=0 recent=0 uidnext=1 uidvalidity=1577952633 unseen=0 highestmodseq=1 vsize=0 guid=c92f64f79f0d1ed01e6d5b314f04886c firstsaved=never
Any ideas what else to try or how to debug that further?
I can send you the full log to your personal address, if that helps ...
Ralf You should rm -rf the target folder first. Can you attach
doveadm -D backuplogs? Check that it won't contain passwords. The mailbox directory did NOT exist before, therefore no need to rm -rf it.I send the logs to your private address only, I feel not comfortable to post them on the list.
Ralf
You did not delete the mailbox.
You are using **mbox** for subscription namespace, please don't.
Also
Nov 03 09:05:33 dsync(): Debug: brain M: Local mailbox tree: INBOX guid=c8adef115c84636335000000effb6190 uid_validity=1667466332 uid_next=1 subs=no last_change=0 last_subs=0 Nov 03 09:05:33 dsync(): Debug: brain S: Local mailbox tree: INBOX guid=c92f64f79f0d1ed01e6d5b314f04886c uid_validity=1577952633 uid_next=32746 subs=no last_change=0 last_subs=0
This clearly shows that you did not, in fact, rm -rf the user's mailboxes prir running backup. Can you please try again, and clean up the user from t target mailsystem before running backup again?
I would also strongly recommend not using ACL plugin while doing backup, unless you are backing up ACLs from source system.
I removed all ACL plugin and config from my Dovecot config and removed the subscription namespace from my doveadm backup command, but now it fails with an error:
doveadm -o imapc_user='someuser' -o imapc_password='secret' -D backup -n INBOX/ -R -u someuser@somedomain imapc:
Nov 03 11:12:15 doveadm(someuser@somedomain 156): Debug: Effective uid=90, gid=101, home=/var/dovecot/imap/somedomain/someuser Nov 03 11:12:15 doveadm(someuser@somedomain 156): Debug: Home dir not found: /var/dovecot/imap/somedomain/someuser
Nov 03 11:12:15 doveadm(someuser@somedomain): Error: namespace configuration error: subscriptions=yes namespace missing
doveadm config -n is attached.
Ralf
-- Ralf Becker EGroupware GmbH [www.egroupware.org] Handelsregister HRB Kaiserslautern 3587 Geschäftsführer Birgit und Ralf Becker Leibnizstr. 17, 67663 Kaiserslautern, Germany Telefon +49 631 31657-0
On 03/11/2022 13:19 EET Ralf Becker rb@egroupware.org wrote:
Hi Aki,
Am 03.11.22 um 10:54 schrieb Aki Tuomi:
On 03/11/2022 11:46 EET Ralf Becker rb@egroupware.org wrote:
Hi Aki,
Am 03.11.22 um 10:29 schrieb Aki Tuomi:
On 03/11/2022 11:27 EET Ralf Becker rb@egroupware.org wrote:
Hi Aki,
Am 03.11.22 um 09:12 schrieb Aki Tuomi:
> On 03/11/2022 10:09 EET Ralf Becker rb@egroupware.org wrote: > >
> Hi Aki, > > Am 03.11.22 um 08:50 schrieb Aki Tuomi: >>> On 03/11/2022 09:46 EET Ralf Becker rb@egroupware.org wrote: >>> >>>
>>> I'm trying to migrate an old Cyrus 2.5 server to Dovecot 2.3.19 using >>> doveadm backup -R, which works for all folders but the INBOX itself, >>> which always stays empty. >>> >>> The Cyrus side uses altnamespace:no and unixhierarchysep:no, it's used >>> as imapc: remote in doveadm backup -R with imapc_list_prefix=INBOX >>> >>> Dovecot uses the following namespace to migrate into: >>> >>> namespace inboxes { >>> inbox = yes >>> location = >>> mailbox Sent { >>> auto = subscribe >>> special_use = \Sent >>> } >>> ### some more folders omitted ### >>> prefix = INBOX/ >>> separator = / >>> subscriptions = no >>> } >> Hi! >> >> When syncing mailboxes from other server, you should use migration config file, which has **no** auto=subscribe or auto=create folders, as these can mess up with synchronization. >> >> Please see https://doc.dovecot.org/admin_manual/migrating_mailboxes/ for more details. > Does a migration config file specified with doveadm -c <file> add to and > overwrite the existing Dovecot configuration for the time the command > runs, like the -o options, or do I need to start a separate server with > a full configuration to e.g. have my authentication and mailbox location > available? > > Ralf > It does not add/replace/overwrite configuration, you provide a fresh config file which is used *instead of* the default dovecot.conf.You don't need to run a separate instance necessarely, although in some larger migrations this has been used as well. I created now a separate instance with a modified configuration file with no auto=subscribe (or create), no replication and an empty storage. doveadm config -n is attached.
Unfortunately the result is identical to my previous tries:
doveadm -o namespace/subs/location=mbox:/var/dovecot/subs -o imapc_user='someuser' -o imapc_password='secret' -D backup -n INBOX/ -R -u someuser@somedomain imapc: 2>&1 | tee /tmp/doveadm-backup.log
Nov 03 09:06:35 dsync(someuser@somedomain): Warning: Mailbox changes caused a desync. You may want to run dsync again: Remote lost mailbox GUID c92f64f79f0d1ed01e6d5b314f04886c (maybe it was just deleted?)
doveadm mailbox status -u someuser@somedomain all INBOX INBOX messages=0 recent=0 uidnext=1 uidvalidity=1577952633 unseen=0 highestmodseq=1 vsize=0 guid=c92f64f79f0d1ed01e6d5b314f04886c firstsaved=never
Any ideas what else to try or how to debug that further?
I can send you the full log to your personal address, if that helps ...
Ralf You should rm -rf the target folder first. Can you attach
doveadm -D backuplogs? Check that it won't contain passwords. The mailbox directory did NOT exist before, therefore no need to rm -rf it.I send the logs to your private address only, I feel not comfortable to post them on the list.
Ralf
You did not delete the mailbox.
You are using **mbox** for subscription namespace, please don't.
Also
Nov 03 09:05:33 dsync(): Debug: brain M: Local mailbox tree: INBOX guid=c8adef115c84636335000000effb6190 uid_validity=1667466332 uid_next=1 subs=no last_change=0 last_subs=0 Nov 03 09:05:33 dsync(): Debug: brain S: Local mailbox tree: INBOX guid=c92f64f79f0d1ed01e6d5b314f04886c uid_validity=1577952633 uid_next=32746 subs=no last_change=0 last_subs=0
This clearly shows that you did not, in fact, rm -rf the user's mailboxes prir running backup. Can you please try again, and clean up the user from t target mailsystem before running backup again?
I would also strongly recommend not using ACL plugin while doing backup, unless you are backing up ACLs from source system.
I removed all ACL plugin and config from my Dovecot config and removed the subscription namespace from my doveadm backup command, but now it fails with an error:
doveadm -o imapc_user='someuser' -o imapc_password='secret' -D backup -n INBOX/ -R -u someuser@somedomain imapc:
Nov 03 11:12:15 doveadm(someuser@somedomain 156): Debug: Effective uid=90, gid=101, home=/var/dovecot/imap/somedomain/someuser Nov 03 11:12:15 doveadm(someuser@somedomain 156): Debug: Home dir not found: /var/dovecot/imap/somedomain/someuser
Nov 03 11:12:15 doveadm(someuser@somedomain): Error: namespace configuration error: subscriptions=yes namespace missing
doveadm config -n is attached.
Ralf
You can keep the subscription namespace in your config, otherwise you have no place to store subscriptions into. Just don't use mbox driver for it, use sdbox instead.
namespace subscriptions { hidden = yes list = no location = sdbox:/index/directory/%n/subs-shared/:LISTINDEX=root:SUBSCRIPTIONS=subscriptions-shared:LAYOUT=INDEX prefix = separator = / subscriptions = yes }
Aki
On 03/11/2022 13:23 EET Aki Tuomi aki.tuomi@open-xchange.com wrote:
On 03/11/2022 13:19 EET Ralf Becker rb@egroupware.org wrote:
Hi Aki,
Am 03.11.22 um 10:54 schrieb Aki Tuomi:
On 03/11/2022 11:46 EET Ralf Becker rb@egroupware.org wrote:
Hi Aki,
Am 03.11.22 um 10:29 schrieb Aki Tuomi:
On 03/11/2022 11:27 EET Ralf Becker rb@egroupware.org wrote:
Hi Aki,
Am 03.11.22 um 09:12 schrieb Aki Tuomi: >> On 03/11/2022 10:09 EET Ralf Becker rb@egroupware.org wrote: >> >>
>> Hi Aki, >> >> Am 03.11.22 um 08:50 schrieb Aki Tuomi: >>>> On 03/11/2022 09:46 EET Ralf Becker rb@egroupware.org wrote: >>>> >>>>
>>>> I'm trying to migrate an old Cyrus 2.5 server to Dovecot 2.3.19 using >>>> doveadm backup -R, which works for all folders but the INBOX itself, >>>> which always stays empty. >>>> >>>> The Cyrus side uses altnamespace:no and unixhierarchysep:no, it's used >>>> as imapc: remote in doveadm backup -R with imapc_list_prefix=INBOX >>>> >>>> Dovecot uses the following namespace to migrate into: >>>> >>>> namespace inboxes { >>>> inbox = yes >>>> location = >>>> mailbox Sent { >>>> auto = subscribe >>>> special_use = \Sent >>>> } >>>> ### some more folders omitted ### >>>> prefix = INBOX/ >>>> separator = / >>>> subscriptions = no >>>> } >>> Hi! >>> >>> When syncing mailboxes from other server, you should use migration config file, which has **no** auto=subscribe or auto=create folders, as these can mess up with synchronization. >>> >>> Please see https://doc.dovecot.org/admin_manual/migrating_mailboxes/ for more details. >> Does a migration config file specified with doveadm -c <file> add to and >> overwrite the existing Dovecot configuration for the time the command >> runs, like the -o options, or do I need to start a separate server with >> a full configuration to e.g. have my authentication and mailbox location >> available? >> >> Ralf >> > It does not add/replace/overwrite configuration, you provide a fresh config file which is used *instead of* the default dovecot.conf. > > You don't need to run a separate instance necessarely, although in some larger migrations this has been used as well. I created now a separate instance with a modified configuration file with no auto=subscribe (or create), no replication and an empty storage. doveadm config -n is attached.Unfortunately the result is identical to my previous tries:
doveadm -o namespace/subs/location=mbox:/var/dovecot/subs -o imapc_user='someuser' -o imapc_password='secret' -D backup -n INBOX/ -R -u someuser@somedomain imapc: 2>&1 | tee /tmp/doveadm-backup.log
Nov 03 09:06:35 dsync(someuser@somedomain): Warning: Mailbox changes caused a desync. You may want to run dsync again: Remote lost mailbox GUID c92f64f79f0d1ed01e6d5b314f04886c (maybe it was just deleted?)
doveadm mailbox status -u someuser@somedomain all INBOX INBOX messages=0 recent=0 uidnext=1 uidvalidity=1577952633 unseen=0 highestmodseq=1 vsize=0 guid=c92f64f79f0d1ed01e6d5b314f04886c firstsaved=never
Any ideas what else to try or how to debug that further?
I can send you the full log to your personal address, if that helps ...
Ralf You should rm -rf the target folder first. Can you attach
doveadm -D backuplogs? Check that it won't contain passwords. The mailbox directory did NOT exist before, therefore no need to rm -rf it.I send the logs to your private address only, I feel not comfortable to post them on the list.
Ralf
You did not delete the mailbox.
You are using **mbox** for subscription namespace, please don't.
Also
Nov 03 09:05:33 dsync(): Debug: brain M: Local mailbox tree: INBOX guid=c8adef115c84636335000000effb6190 uid_validity=1667466332 uid_next=1 subs=no last_change=0 last_subs=0 Nov 03 09:05:33 dsync(): Debug: brain S: Local mailbox tree: INBOX guid=c92f64f79f0d1ed01e6d5b314f04886c uid_validity=1577952633 uid_next=32746 subs=no last_change=0 last_subs=0
This clearly shows that you did not, in fact, rm -rf the user's mailboxes prir running backup. Can you please try again, and clean up the user from t target mailsystem before running backup again?
I would also strongly recommend not using ACL plugin while doing backup, unless you are backing up ACLs from source system.
I removed all ACL plugin and config from my Dovecot config and removed the subscription namespace from my doveadm backup command, but now it fails with an error:
doveadm -o imapc_user='someuser' -o imapc_password='secret' -D backup -n INBOX/ -R -u someuser@somedomain imapc:
Nov 03 11:12:15 doveadm(someuser@somedomain 156): Debug: Effective uid=90, gid=101, home=/var/dovecot/imap/somedomain/someuser Nov 03 11:12:15 doveadm(someuser@somedomain 156): Debug: Home dir not found: /var/dovecot/imap/somedomain/someuser
Nov 03 11:12:15 doveadm(someuser@somedomain): Error: namespace configuration error: subscriptions=yes namespace missing
doveadm config -n is attached.
Ralf
You can keep the subscription namespace in your config, otherwise you have no place to store subscriptions into. Just don't use mbox driver for it, use sdbox instead.
namespace subscriptions { hidden = yes list = no location = sdbox:/index/directory/%n/subs-shared/:LISTINDEX=root:SUBSCRIPTIONS=subscriptions-shared:LAYOUT=INDEX prefix = separator = / subscriptions = yes }
Aki
Oh and forgot to mention, for INBOX namespace you can leave subscriptions turned on. This is only needed for shared/public namespaces in any case..
Aki
Hi Aki,
Am 03.11.22 um 12:27 schrieb Aki Tuomi:
On 03/11/2022 13:23 EET Aki Tuomi aki.tuomi@open-xchange.com wrote:
On 03/11/2022 13:19 EET Ralf Becker rb@egroupware.org wrote:
Hi Aki,
On 03/11/2022 11:46 EET Ralf Becker rb@egroupware.org wrote:
Hi Aki,
Am 03.11.22 um 10:29 schrieb Aki Tuomi:
> On 03/11/2022 11:27 EET Ralf Becker rb@egroupware.org wrote: > >
> Hi Aki, > > Am 03.11.22 um 09:12 schrieb Aki Tuomi: >>> On 03/11/2022 10:09 EET Ralf Becker rb@egroupware.org wrote: >>> >>>
>>> Hi Aki, >>> >>> Am 03.11.22 um 08:50 schrieb Aki Tuomi: >>>>> On 03/11/2022 09:46 EET Ralf Becker rb@egroupware.org wrote: >>>>> >>>>>
>>>>> I'm trying to migrate an old Cyrus 2.5 server to Dovecot 2.3.19 using >>>>> doveadm backup -R, which works for all folders but the INBOX itself, >>>>> which always stays empty. >>>>> >>>>> The Cyrus side uses altnamespace:no and unixhierarchysep:no, it's used >>>>> as imapc: remote in doveadm backup -R with imapc_list_prefix=INBOX >>>>> >>>>> Dovecot uses the following namespace to migrate into: >>>>> >>>>> namespace inboxes { >>>>> inbox = yes >>>>> location = >>>>> mailbox Sent { >>>>> auto = subscribe >>>>> special_use = \Sent >>>>> } >>>>> ### some more folders omitted ### >>>>> prefix = INBOX/ >>>>> separator = / >>>>> subscriptions = no >>>>> } >>>> Hi! >>>> >>>> When syncing mailboxes from other server, you should use migration config file, which has **no** auto=subscribe or auto=create folders, as these can mess up with synchronization. >>>> >>>> Please see https://doc.dovecot.org/admin_manual/migrating_mailboxes/ for more details. >>> Does a migration config file specified with doveadm -c <file> add to and >>> overwrite the existing Dovecot configuration for the time the command >>> runs, like the -o options, or do I need to start a separate server with >>> a full configuration to e.g. have my authentication and mailbox location >>> available? >>> >>> Ralf >>> >> It does not add/replace/overwrite configuration, you provide a fresh config file which is used *instead of* the default dovecot.conf. >> >> You don't need to run a separate instance necessarely, although in some larger migrations this has been used as well. > I created now a separate instance with a modified configuration file > with no auto=subscribe (or create), no replication and an empty storage. > doveadm config -n is attached. > > Unfortunately the result is identical to my previous tries: > > doveadm -o namespace/subs/location=mbox:/var/dovecot/subs -o > imapc_user='someuser' -o imapc_password='secret' -D backup -n INBOX/ -R > -u someuser@somedomain imapc: 2>&1 | tee /tmp/doveadm-backup.log > > Nov 03 09:06:35 dsync(someuser@somedomain): Warning: Mailbox changes > caused a desync. You may want to run dsync again: Remote lost mailbox > GUID c92f64f79f0d1ed01e6d5b314f04886c (maybe it was just deleted?) > > doveadm mailbox status -u someuser@somedomain all INBOX > INBOX messages=0 recent=0 uidnext=1 uidvalidity=1577952633 unseen=0 > highestmodseq=1 vsize=0 guid=c92f64f79f0d1ed01e6d5b314f04886c > firstsaved=never > > Any ideas what else to try or how to debug that further? > > I can send you the full log to your personal address, if that helps ... > > Ralf You should rm -rf the target folder first. Can you attachdoveadm -D backuplogs? Check that it won't contain passwords. The mailbox directory did NOT exist before, therefore no need to rm -rf it.I send the logs to your private address only, I feel not comfortable to post them on the list.
Ralf
You did not delete the mailbox.
You are using **mbox** for subscription namespace, please don't.
Also
Nov 03 09:05:33 dsync(): Debug: brain M: Local mailbox tree: INBOX guid=c8adef115c84636335000000effb6190 uid_validity=1667466332 uid_next=1 subs=no last_change=0 last_subs=0 Nov 03 09:05:33 dsync(): Debug: brain S: Local mailbox tree: INBOX guid=c92f64f79f0d1ed01e6d5b314f04886c uid_validity=1577952633 uid_next=32746 subs=no last_change=0 last_subs=0
This clearly shows that you did not, in fact, rm -rf the user's mailboxes prir running backup. Can you please try again, and clean up the user from t target mailsystem before running backup again?
I would also strongly recommend not using ACL plugin while doing backup, unless you are backing up ACLs from source system. I removed all ACL plugin and config from my Dovecot config and removed
Am 03.11.22 um 10:54 schrieb Aki Tuomi: the subscription namespace from my doveadm backup command, but now it fails with an error:
doveadm -o imapc_user='someuser' -o imapc_password='secret' -D backup -n INBOX/ -R -u someuser@somedomain imapc:
Nov 03 11:12:15 doveadm(someuser@somedomain 156): Debug: Effective uid=90, gid=101, home=/var/dovecot/imap/somedomain/someuser Nov 03 11:12:15 doveadm(someuser@somedomain 156): Debug: Home dir not found: /var/dovecot/imap/somedomain/someuser
Nov 03 11:12:15 doveadm(someuser@somedomain): Error: namespace configuration error: subscriptions=yes namespace missing
doveadm config -n is attached.
Ralf You can keep the subscription namespace in your config, otherwise you have no place to store subscriptions into. Just don't use mbox driver for it, use sdbox instead.
namespace subscriptions { hidden = yes list = no location = sdbox:/index/directory/%n/subs-shared/:LISTINDEX=root:SUBSCRIPTIONS=subscriptions-shared:LAYOUT=INDEX prefix = separator = / subscriptions = yes }
Aki Oh and forgot to mention, for INBOX namespace you can leave subscriptions turned on. This is only needed for shared/public namespaces in any case..
I changed the config now, as you suggested (doveadm config -n is attached), unfortunately the result stays the same:
doveadm -o imapc_user='someuser' -o imapc_password='secret' -D backup -n INBOX/ -R -u someuser@somedomain imapc: 2>&1|tee /tmp/doveadm-backup-someuser2.log
Nov 03 11:35:39 dsync(someuser@somedomain): Warning: Mailbox changes caused a desync. You may want to run dsync again: Remote lost mailbox GUID c92f64f79f0d1ed01e6d5b314f04886c (maybe it was just deleted?)
doveadm mailbox status -u someuser@somedomain all INBOX INBOX messages=0 recent=0 uidnext=1 uidvalidity=1577952633 unseen=0 highestmodseq=1 vsize=0 guid=c92f64f79f0d1ed01e6d5b314f04886c firstsaved=never
I send the full log again to your private address.
Ralf
-- Ralf Becker EGroupware GmbH [www.egroupware.org] Handelsregister HRB Kaiserslautern 3587 Geschäftsführer Birgit und Ralf Becker Leibnizstr. 17, 67663 Kaiserslautern, Germany Telefon +49 631 31657-0
On 03/11/2022 13:43 EET Ralf Becker rb@egroupware.org wrote:
Hi Aki,
Am 03.11.22 um 12:27 schrieb Aki Tuomi:
On 03/11/2022 13:23 EET Aki Tuomi aki.tuomi@open-xchange.com wrote:
On 03/11/2022 13:19 EET Ralf Becker rb@egroupware.org wrote:
Hi Aki,
On 03/11/2022 11:46 EET Ralf Becker rb@egroupware.org wrote:
Hi Aki,
Am 03.11.22 um 10:29 schrieb Aki Tuomi: >> On 03/11/2022 11:27 EET Ralf Becker rb@egroupware.org wrote: >> >>
>> Hi Aki, >> >> Am 03.11.22 um 09:12 schrieb Aki Tuomi: >>>> On 03/11/2022 10:09 EET Ralf Becker rb@egroupware.org wrote: >>>> >>>>
>>>> Hi Aki, >>>> >>>> Am 03.11.22 um 08:50 schrieb Aki Tuomi: >>>>>> On 03/11/2022 09:46 EET Ralf Becker rb@egroupware.org wrote: >>>>>> >>>>>>
>>>>>> I'm trying to migrate an old Cyrus 2.5 server to Dovecot 2.3.19 using >>>>>> doveadm backup -R, which works for all folders but the INBOX itself, >>>>>> which always stays empty. >>>>>> >>>>>> The Cyrus side uses altnamespace:no and unixhierarchysep:no, it's used >>>>>> as imapc: remote in doveadm backup -R with imapc_list_prefix=INBOX >>>>>> >>>>>> Dovecot uses the following namespace to migrate into: >>>>>> >>>>>> namespace inboxes { >>>>>> inbox = yes >>>>>> location = >>>>>> mailbox Sent { >>>>>> auto = subscribe >>>>>> special_use = \Sent >>>>>> } >>>>>> ### some more folders omitted ### >>>>>> prefix = INBOX/ >>>>>> separator = / >>>>>> subscriptions = no >>>>>> } >>>>> Hi! >>>>> >>>>> When syncing mailboxes from other server, you should use migration config file, which has **no** auto=subscribe or auto=create folders, as these can mess up with synchronization. >>>>> >>>>> Please see https://doc.dovecot.org/admin_manual/migrating_mailboxes/ for more details. >>>> Does a migration config file specified with doveadm -c <file> add to and >>>> overwrite the existing Dovecot configuration for the time the command >>>> runs, like the -o options, or do I need to start a separate server with >>>> a full configuration to e.g. have my authentication and mailbox location >>>> available? >>>> >>>> Ralf >>>> >>> It does not add/replace/overwrite configuration, you provide a fresh config file which is used *instead of* the default dovecot.conf. >>> >>> You don't need to run a separate instance necessarely, although in some larger migrations this has been used as well. >> I created now a separate instance with a modified configuration file >> with no auto=subscribe (or create), no replication and an empty storage. >> doveadm config -n is attached. >> >> Unfortunately the result is identical to my previous tries: >> >> doveadm -o namespace/subs/location=mbox:/var/dovecot/subs -o >> imapc_user='someuser' -o imapc_password='secret' -D backup -n INBOX/ -R >> -u someuser@somedomain imapc: 2>&1 | tee /tmp/doveadm-backup.log >> >> Nov 03 09:06:35 dsync(someuser@somedomain): Warning: Mailbox changes >> caused a desync. You may want to run dsync again: Remote lost mailbox >> GUID c92f64f79f0d1ed01e6d5b314f04886c (maybe it was just deleted?) >> >> doveadm mailbox status -u someuser@somedomain all INBOX >> INBOX messages=0 recent=0 uidnext=1 uidvalidity=1577952633 unseen=0 >> highestmodseq=1 vsize=0 guid=c92f64f79f0d1ed01e6d5b314f04886c >> firstsaved=never >> >> Any ideas what else to try or how to debug that further? >> >> I can send you the full log to your personal address, if that helps ... >> >> Ralf > You should rm -rf the target folder first. Can you attachdoveadm -D backuplogs? Check that it won't contain passwords. The mailbox directory did NOT exist before, therefore no need to rm -rf it.I send the logs to your private address only, I feel not comfortable to post them on the list.
Ralf
You did not delete the mailbox.
You are using **mbox** for subscription namespace, please don't.
Also
Nov 03 09:05:33 dsync(): Debug: brain M: Local mailbox tree: INBOX guid=c8adef115c84636335000000effb6190 uid_validity=1667466332 uid_next=1 subs=no last_change=0 last_subs=0 Nov 03 09:05:33 dsync(): Debug: brain S: Local mailbox tree: INBOX guid=c92f64f79f0d1ed01e6d5b314f04886c uid_validity=1577952633 uid_next=32746 subs=no last_change=0 last_subs=0
This clearly shows that you did not, in fact, rm -rf the user's mailboxes prir running backup. Can you please try again, and clean up the user from t target mailsystem before running backup again?
I would also strongly recommend not using ACL plugin while doing backup, unless you are backing up ACLs from source system. I removed all ACL plugin and config from my Dovecot config and removed
Am 03.11.22 um 10:54 schrieb Aki Tuomi: the subscription namespace from my doveadm backup command, but now it fails with an error:
doveadm -o imapc_user='someuser' -o imapc_password='secret' -D backup -n INBOX/ -R -u someuser@somedomain imapc:
Nov 03 11:12:15 doveadm(someuser@somedomain 156): Debug: Effective uid=90, gid=101, home=/var/dovecot/imap/somedomain/someuser Nov 03 11:12:15 doveadm(someuser@somedomain 156): Debug: Home dir not found: /var/dovecot/imap/somedomain/someuser
Nov 03 11:12:15 doveadm(someuser@somedomain): Error: namespace configuration error: subscriptions=yes namespace missing
doveadm config -n is attached.
Ralf You can keep the subscription namespace in your config, otherwise you have no place to store subscriptions into. Just don't use mbox driver for it, use sdbox instead.
namespace subscriptions { hidden = yes list = no location = sdbox:/index/directory/%n/subs-shared/:LISTINDEX=root:SUBSCRIPTIONS=subscriptions-shared:LAYOUT=INDEX prefix = separator = / subscriptions = yes }
Aki Oh and forgot to mention, for INBOX namespace you can leave subscriptions turned on. This is only needed for shared/public namespaces in any case..
I changed the config now, as you suggested (doveadm config -n is attached), unfortunately the result stays the same:
doveadm -o imapc_user='someuser' -o imapc_password='secret' -D backup -n INBOX/ -R -u someuser@somedomain imapc: 2>&1|tee /tmp/doveadm-backup-someuser2.log
Nov 03 11:35:39 dsync(someuser@somedomain): Warning: Mailbox changes caused a desync. You may want to run dsync again: Remote lost mailbox GUID c92f64f79f0d1ed01e6d5b314f04886c (maybe it was just deleted?)
doveadm mailbox status -u someuser@somedomain all INBOX INBOX messages=0 recent=0 uidnext=1 uidvalidity=1577952633 unseen=0 highestmodseq=1 vsize=0 guid=c92f64f79f0d1ed01e6d5b314f04886c firstsaved=never
I send the full log again to your private address.
Ralf
Seems I can reproduce this issue, we'll look into this.
Aki
Am 03.11.22 um 14:29 schrieb Aki Tuomi:
On 03/11/2022 13:43 EET Ralf Becker rb@egroupware.org wrote:
Hi Aki,
Am 03.11.22 um 12:27 schrieb Aki Tuomi:
On 03/11/2022 13:23 EET Aki Tuomi aki.tuomi@open-xchange.com wrote:
On 03/11/2022 13:19 EET Ralf Becker rb@egroupware.org wrote:
Hi Aki,
> On 03/11/2022 11:46 EET Ralf Becker rb@egroupware.org wrote: > >
> Hi Aki, > > Am 03.11.22 um 10:29 schrieb Aki Tuomi: >>> On 03/11/2022 11:27 EET Ralf Becker rb@egroupware.org wrote: >>> >>>
>>> Hi Aki, >>> >>> Am 03.11.22 um 09:12 schrieb Aki Tuomi: >>>>> On 03/11/2022 10:09 EET Ralf Becker rb@egroupware.org wrote: >>>>> >>>>>
>>>>> Hi Aki, >>>>> >>>>> Am 03.11.22 um 08:50 schrieb Aki Tuomi: >>>>>>> On 03/11/2022 09:46 EET Ralf Becker rb@egroupware.org wrote: >>>>>>> >>>>>>>
>>>>>>> I'm trying to migrate an old Cyrus 2.5 server to Dovecot 2.3.19 using >>>>>>> doveadm backup -R, which works for all folders but the INBOX itself, >>>>>>> which always stays empty. >>>>>>> >>>>>>> The Cyrus side uses altnamespace:no and unixhierarchysep:no, it's used >>>>>>> as imapc: remote in doveadm backup -R with imapc_list_prefix=INBOX >>>>>>> >>>>>>> Dovecot uses the following namespace to migrate into: >>>>>>> >>>>>>> namespace inboxes { >>>>>>> inbox = yes >>>>>>> location = >>>>>>> mailbox Sent { >>>>>>> auto = subscribe >>>>>>> special_use = \Sent >>>>>>> } >>>>>>> ### some more folders omitted ### >>>>>>> prefix = INBOX/ >>>>>>> separator = / >>>>>>> subscriptions = no >>>>>>> } >>>>>> Hi! >>>>>> >>>>>> When syncing mailboxes from other server, you should use migration config file, which has **no** auto=subscribe or auto=create folders, as these can mess up with synchronization. >>>>>> >>>>>> Please see https://doc.dovecot.org/admin_manual/migrating_mailboxes/ for more details. >>>>> Does a migration config file specified with doveadm -c <file> add to and >>>>> overwrite the existing Dovecot configuration for the time the command >>>>> runs, like the -o options, or do I need to start a separate server with >>>>> a full configuration to e.g. have my authentication and mailbox location >>>>> available? >>>>> >>>>> Ralf >>>>> >>>> It does not add/replace/overwrite configuration, you provide a fresh config file which is used *instead of* the default dovecot.conf. >>>> >>>> You don't need to run a separate instance necessarely, although in some larger migrations this has been used as well. >>> I created now a separate instance with a modified configuration file >>> with no auto=subscribe (or create), no replication and an empty storage. >>> doveadm config -n is attached. >>> >>> Unfortunately the result is identical to my previous tries: >>> >>> doveadm -o namespace/subs/location=mbox:/var/dovecot/subs -o >>> imapc_user='someuser' -o imapc_password='secret' -D backup -n INBOX/ -R >>> -u someuser@somedomain imapc: 2>&1 | tee /tmp/doveadm-backup.log >>> >>> Nov 03 09:06:35 dsync(someuser@somedomain): Warning: Mailbox changes >>> caused a desync. You may want to run dsync again: Remote lost mailbox >>> GUID c92f64f79f0d1ed01e6d5b314f04886c (maybe it was just deleted?) >>> >>> doveadm mailbox status -u someuser@somedomain all INBOX >>> INBOX messages=0 recent=0 uidnext=1 uidvalidity=1577952633 unseen=0 >>> highestmodseq=1 vsize=0 guid=c92f64f79f0d1ed01e6d5b314f04886c >>> firstsaved=never >>> >>> Any ideas what else to try or how to debug that further? >>> >>> I can send you the full log to your personal address, if that helps ... >>> >>> Ralf >> You should rm -rf the target folder first. Can you attachdoveadm -D backuplogs? Check that it won't contain passwords. > The mailbox directory did NOT exist before, therefore no need to rm -rf it. > > I send the logs to your private address only, I feel not comfortable to > post them on the list. > > Ralf >
You did not delete the mailbox.
You are using **mbox** for subscription namespace, please don't.
Also
Nov 03 09:05:33 dsync(): Debug: brain M: Local mailbox tree: INBOX guid=c8adef115c84636335000000effb6190 uid_validity=1667466332 uid_next=1 subs=no last_change=0 last_subs=0 Nov 03 09:05:33 dsync(): Debug: brain S: Local mailbox tree: INBOX guid=c92f64f79f0d1ed01e6d5b314f04886c uid_validity=1577952633 uid_next=32746 subs=no last_change=0 last_subs=0
This clearly shows that you did not, in fact, rm -rf the user's mailboxes prir running backup. Can you please try again, and clean up the user from t target mailsystem before running backup again?
I would also strongly recommend not using ACL plugin while doing backup, unless you are backing up ACLs from source system. I removed all ACL plugin and config from my Dovecot config and removed
Am 03.11.22 um 10:54 schrieb Aki Tuomi: the subscription namespace from my doveadm backup command, but now it fails with an error:
doveadm -o imapc_user='someuser' -o imapc_password='secret' -D backup -n INBOX/ -R -u someuser@somedomain imapc:
Nov 03 11:12:15 doveadm(someuser@somedomain 156): Debug: Effective uid=90, gid=101, home=/var/dovecot/imap/somedomain/someuser Nov 03 11:12:15 doveadm(someuser@somedomain 156): Debug: Home dir not found: /var/dovecot/imap/somedomain/someuser
Nov 03 11:12:15 doveadm(someuser@somedomain): Error: namespace configuration error: subscriptions=yes namespace missing
doveadm config -n is attached.
Ralf You can keep the subscription namespace in your config, otherwise you have no place to store subscriptions into. Just don't use mbox driver for it, use sdbox instead.
namespace subscriptions { hidden = yes list = no location = sdbox:/index/directory/%n/subs-shared/:LISTINDEX=root:SUBSCRIPTIONS=subscriptions-shared:LAYOUT=INDEX prefix = separator = / subscriptions = yes }
Aki Oh and forgot to mention, for INBOX namespace you can leave subscriptions turned on. This is only needed for shared/public namespaces in any case.. I changed the config now, as you suggested (doveadm config -n is attached), unfortunately the result stays the same:
doveadm -o imapc_user='someuser' -o imapc_password='secret' -D backup -n INBOX/ -R -u someuser@somedomain imapc: 2>&1|tee /tmp/doveadm-backup-someuser2.log
Nov 03 11:35:39 dsync(someuser@somedomain): Warning: Mailbox changes caused a desync. You may want to run dsync again: Remote lost mailbox GUID c92f64f79f0d1ed01e6d5b314f04886c (maybe it was just deleted?)
doveadm mailbox status -u someuser@somedomain all INBOX INBOX messages=0 recent=0 uidnext=1 uidvalidity=1577952633 unseen=0 highestmodseq=1 vsize=0 guid=c92f64f79f0d1ed01e6d5b314f04886c firstsaved=never
I send the full log again to your private address.
Ralf
Seems I can reproduce this issue, we'll look into this.
Aki
Thanks Aki :)
Please reply to this release, if you have any news, preferably with a commit, so I can test ...
Ralf
-- Ralf Becker EGroupware GmbH [www.egroupware.org] Handelsregister HRB Kaiserslautern 3587 Geschäftsführer Birgit und Ralf Becker Leibnizstr. 17, 67663 Kaiserslautern, Germany Telefon +49 631 31657-0
I send the full log again to your private address.
Ralf
Seems I can reproduce this issue, we'll look into this.
Aki
Hi!
Can you try if
https://github.com/dovecot/core/compare/a485443e00a7e1b93ed9b7d065c89fcd2eb9...
fixes your issue?
Aki
Hi Aki,
Am 08.11.22 um 10:07 schrieb Aki Tuomi:
I send the full log again to your private address.
Ralf Seems I can reproduce this issue, we'll look into this.
Aki Hi!
Can you try if
https://github.com/dovecot/core/compare/a485443e00a7e1b93ed9b7d065c89fcd2eb9...
fixes your issue?
I will try and come back to you.
I currently have no build environment for Dovecot and change the migration to use imapsync instead of doveadm sync|backup.
Ralf
-- Ralf Becker EGroupware GmbH [www.egroupware.org] Handelsregister HRB Kaiserslautern 3587 Geschäftsführer Birgit und Ralf Becker Leibnizstr. 17, 67663 Kaiserslautern, Germany Telefon +49 631 31657-0
Hi Aki,
Am 08.11.22 um 12:02 schrieb Ralf Becker:
Hi Aki,
Am 08.11.22 um 10:07 schrieb Aki Tuomi:
I send the full log again to your private address.
Ralf Seems I can reproduce this issue, we'll look into this.
Aki Hi!
Can you try if
https://github.com/dovecot/core/compare/a485443e00a7e1b93ed9b7d065c89fcd2eb9...
fixes your issue?
I will try and come back to you.
I currently have no build environment for Dovecot and change the migration to use imapsync instead of doveadm sync|backup.
Ralf
Do you by chance build nightly containers or have a Docker file building Dovecot from the git sources?
That would make testing things like that a lot easier :)
Ralf
-- Ralf Becker EGroupware GmbH [www.egroupware.org] Handelsregister HRB Kaiserslautern 3587 Geschäftsführer Birgit und Ralf Becker Leibnizstr. 17, 67663 Kaiserslautern, Germany Telefon +49 631 31657-0
participants (3)
-
Aki Tuomi
-
Paul Kudla
-
Ralf Becker