doveadm backup|sync works for every folder but INBOX

Thu Nov 3 11:27:43 UTC 2022

> On 03/11/2022 13:23 EET Aki Tuomi <aki.tuomi at open-xchange.com> wrote:
> 
>  
> > On 03/11/2022 13:19 EET Ralf Becker <rb at egroupware.org> wrote:
> > 
> >  
> > Hi Aki,
> > 
> > Am 03.11.22 um 10:54 schrieb Aki Tuomi:
> > >> On 03/11/2022 11:46 EET Ralf Becker <rb at egroupware.org> wrote:
> > >>
> > >>   
> > >> Hi Aki,
> > >>
> > >> Am 03.11.22 um 10:29 schrieb Aki Tuomi:
> > >>>> On 03/11/2022 11:27 EET Ralf Becker <rb at egroupware.org> wrote:
> > >>>>
> > >>>>    
> > >>>> Hi Aki,
> > >>>>
> > >>>> Am 03.11.22 um 09:12 schrieb Aki Tuomi:
> > >>>>>> On 03/11/2022 10:09 EET Ralf Becker <rb at egroupware.org> wrote:
> > >>>>>>
> > >>>>>>     
> > >>>>>> Hi Aki,
> > >>>>>>
> > >>>>>> Am 03.11.22 um 08:50 schrieb Aki Tuomi:
> > >>>>>>>> On 03/11/2022 09:46 EET Ralf Becker <rb at egroupware.org> wrote:
> > >>>>>>>>
> > >>>>>>>>      
> > >>>>>>>> I'm trying to migrate an old Cyrus 2.5 server to Dovecot 2.3.19 using
> > >>>>>>>> doveadm backup -R, which works for all folders but the INBOX itself,
> > >>>>>>>> which always stays empty.
> > >>>>>>>>
> > >>>>>>>> The Cyrus side uses altnamespace:no and unixhierarchysep:no, it's used
> > >>>>>>>> as imapc: remote in doveadm backup -R with imapc_list_prefix=INBOX
> > >>>>>>>>
> > >>>>>>>> Dovecot uses the following namespace to migrate into:
> > >>>>>>>>
> > >>>>>>>> namespace inboxes {
> > >>>>>>>>        inbox = yes
> > >>>>>>>>        location =
> > >>>>>>>>        mailbox Sent {
> > >>>>>>>>          auto = subscribe
> > >>>>>>>>          special_use = \Sent
> > >>>>>>>>        }
> > >>>>>>>>        ### some more folders omitted ###
> > >>>>>>>>        prefix = INBOX/
> > >>>>>>>>        separator = /
> > >>>>>>>>        subscriptions = no
> > >>>>>>>> }
> > >>>>>>> Hi!
> > >>>>>>>
> > >>>>>>> When syncing mailboxes from other server, you should use migration config file, which has **no** auto=subscribe or auto=create folders, as these can mess up with synchronization.
> > >>>>>>>
> > >>>>>>> Please see https://doc.dovecot.org/admin_manual/migrating_mailboxes/ for more details.
> > >>>>>> Does a migration config file specified with doveadm -c <file> add to and
> > >>>>>> overwrite the existing Dovecot configuration for the time the command
> > >>>>>> runs, like the -o options, or do I need to start a separate server with
> > >>>>>> a full configuration to e.g. have my authentication and mailbox location
> > >>>>>> available?
> > >>>>>>
> > >>>>>> Ralf
> > >>>>>>
> > >>>>> It does not add/replace/overwrite configuration, you provide a fresh config file which is used *instead of* the default dovecot.conf.
> > >>>>>
> > >>>>> You don't need to run a separate instance necessarely, although in some larger migrations this has been used as well.
> > >>>> I created now a separate instance with a modified configuration file
> > >>>> with no auto=subscribe (or create), no replication and an empty storage.
> > >>>> doveadm config -n is attached.
> > >>>>
> > >>>> Unfortunately the result is identical to my previous tries:
> > >>>>
> > >>>> doveadm -o namespace/subs/location=mbox:/var/dovecot/subs -o
> > >>>> imapc_user='someuser' -o imapc_password='secret' -D backup -n INBOX/ -R
> > >>>> -u someuser at somedomain imapc: 2>&1 | tee /tmp/doveadm-backup.log
> > >>>>
> > >>>> Nov 03 09:06:35 dsync(someuser at somedomain): Warning: Mailbox changes
> > >>>> caused a desync. You may want to run dsync again: Remote lost mailbox
> > >>>> GUID c92f64f79f0d1ed01e6d5b314f04886c (maybe it was just deleted?)
> > >>>>
> > >>>> doveadm mailbox status -u someuser at somedomain all INBOX
> > >>>> INBOX messages=0 recent=0 uidnext=1 uidvalidity=1577952633 unseen=0
> > >>>> highestmodseq=1 vsize=0 guid=c92f64f79f0d1ed01e6d5b314f04886c
> > >>>> firstsaved=never
> > >>>>
> > >>>> Any ideas what else to try or how to debug that further?
> > >>>>
> > >>>> I can send you the full log to your personal address, if that helps ...
> > >>>>
> > >>>> Ralf
> > >>> You should rm -rf the target folder first. Can you attach `doveadm -D backup` logs? Check that it won't contain passwords.
> > >> The mailbox directory did NOT exist before, therefore no need to rm -rf it.
> > >>
> > >> I send the logs to your private address only, I feel not comfortable to
> > >> post them on the list.
> > >>
> > >> Ralf
> > >>
> > > 1. You did not delete the mailbox.
> > >
> > > 2. You are using **mbox** for subscription namespace, please don't.
> > >
> > > Also
> > >
> > > Nov 03 09:05:33 dsync(): Debug: brain M: Local mailbox tree: INBOX guid=c8adef115c84636335000000effb6190 uid_validity=1667466332 uid_next=1 subs=no last_change=0 last_subs=0
> > > Nov 03 09:05:33 dsync(): Debug: brain S: Local mailbox tree: INBOX guid=c92f64f79f0d1ed01e6d5b314f04886c uid_validity=1577952633 uid_next=32746 subs=no last_change=0 last_subs=0
> > >
> > > This clearly shows that you did not, in fact, rm -rf the user's mailboxes prir running backup. Can you please try again, and clean up the user from t target mailsystem before running backup again?
> > >
> > > I would also strongly recommend not using ACL plugin while doing backup, unless you are backing up ACLs from source system.
> > 
> > I removed all ACL plugin and config from my Dovecot config and removed 
> > the subscription namespace from my doveadm backup command, but now it 
> > fails with an error:
> > 
> > doveadm -o imapc_user='someuser' -o imapc_password='secret' -D backup -n 
> > INBOX/ -R -u someuser at somedomain imapc:
> > 
> > Nov 03 11:12:15 doveadm(someuser at somedomain 156): Debug: Effective 
> > uid=90, gid=101, home=/var/dovecot/imap/somedomain/someuser
> > Nov 03 11:12:15 doveadm(someuser at somedomain 156): Debug: Home dir not 
> > found: /var/dovecot/imap/somedomain/someuser
> > 
> > Nov 03 11:12:15 doveadm(someuser at somedomain): Error: namespace 
> > configuration error: subscriptions=yes namespace missing
> > 
> > doveadm config -n is attached.
> > 
> > Ralf
> 
> You can keep the subscription namespace in your config, otherwise you have no place to store subscriptions into. Just don't use mbox driver for it, use sdbox instead.
> 
> namespace subscriptions {
>   hidden = yes
>   list = no
>   location = sdbox:/index/directory/%n/subs-shared/:LISTINDEX=root:SUBSCRIPTIONS=subscriptions-shared:LAYOUT=INDEX
>   prefix = 
>   separator = /
>   subscriptions = yes
> }
> 
> Aki

Oh and forgot to mention, for INBOX namespace you can leave subscriptions turned on. This is only needed for shared/public namespaces in any case..

Aki