doveadm backup|sync works for every folder but INBOX

Ralf Becker rb at egroupware.org
Thu Nov 3 11:43:48 UTC 2022


Hi Aki,

Am 03.11.22 um 12:27 schrieb Aki Tuomi:
>> On 03/11/2022 13:23 EET Aki Tuomi <aki.tuomi at open-xchange.com> wrote:
>>> On 03/11/2022 13:19 EET Ralf Becker <rb at egroupware.org> wrote:
>>>
>>>   
>>> Hi Aki,
>>>
>>> Am 03.11.22 um 10:54 schrieb Aki Tuomi:
>>>>> On 03/11/2022 11:46 EET Ralf Becker <rb at egroupware.org> wrote:
>>>>>
>>>>>    
>>>>> Hi Aki,
>>>>>
>>>>> Am 03.11.22 um 10:29 schrieb Aki Tuomi:
>>>>>>> On 03/11/2022 11:27 EET Ralf Becker <rb at egroupware.org> wrote:
>>>>>>>
>>>>>>>     
>>>>>>> Hi Aki,
>>>>>>>
>>>>>>> Am 03.11.22 um 09:12 schrieb Aki Tuomi:
>>>>>>>>> On 03/11/2022 10:09 EET Ralf Becker <rb at egroupware.org> wrote:
>>>>>>>>>
>>>>>>>>>      
>>>>>>>>> Hi Aki,
>>>>>>>>>
>>>>>>>>> Am 03.11.22 um 08:50 schrieb Aki Tuomi:
>>>>>>>>>>> On 03/11/2022 09:46 EET Ralf Becker <rb at egroupware.org> wrote:
>>>>>>>>>>>
>>>>>>>>>>>       
>>>>>>>>>>> I'm trying to migrate an old Cyrus 2.5 server to Dovecot 2.3.19 using
>>>>>>>>>>> doveadm backup -R, which works for all folders but the INBOX itself,
>>>>>>>>>>> which always stays empty.
>>>>>>>>>>>
>>>>>>>>>>> The Cyrus side uses altnamespace:no and unixhierarchysep:no, it's used
>>>>>>>>>>> as imapc: remote in doveadm backup -R with imapc_list_prefix=INBOX
>>>>>>>>>>>
>>>>>>>>>>> Dovecot uses the following namespace to migrate into:
>>>>>>>>>>>
>>>>>>>>>>> namespace inboxes {
>>>>>>>>>>>         inbox = yes
>>>>>>>>>>>         location =
>>>>>>>>>>>         mailbox Sent {
>>>>>>>>>>>           auto = subscribe
>>>>>>>>>>>           special_use = \Sent
>>>>>>>>>>>         }
>>>>>>>>>>>         ### some more folders omitted ###
>>>>>>>>>>>         prefix = INBOX/
>>>>>>>>>>>         separator = /
>>>>>>>>>>>         subscriptions = no
>>>>>>>>>>> }
>>>>>>>>>> Hi!
>>>>>>>>>>
>>>>>>>>>> When syncing mailboxes from other server, you should use migration config file, which has **no** auto=subscribe or auto=create folders, as these can mess up with synchronization.
>>>>>>>>>>
>>>>>>>>>> Please see https://doc.dovecot.org/admin_manual/migrating_mailboxes/ for more details.
>>>>>>>>> Does a migration config file specified with doveadm -c <file> add to and
>>>>>>>>> overwrite the existing Dovecot configuration for the time the command
>>>>>>>>> runs, like the -o options, or do I need to start a separate server with
>>>>>>>>> a full configuration to e.g. have my authentication and mailbox location
>>>>>>>>> available?
>>>>>>>>>
>>>>>>>>> Ralf
>>>>>>>>>
>>>>>>>> It does not add/replace/overwrite configuration, you provide a fresh config file which is used *instead of* the default dovecot.conf.
>>>>>>>>
>>>>>>>> You don't need to run a separate instance necessarely, although in some larger migrations this has been used as well.
>>>>>>> I created now a separate instance with a modified configuration file
>>>>>>> with no auto=subscribe (or create), no replication and an empty storage.
>>>>>>> doveadm config -n is attached.
>>>>>>>
>>>>>>> Unfortunately the result is identical to my previous tries:
>>>>>>>
>>>>>>> doveadm -o namespace/subs/location=mbox:/var/dovecot/subs -o
>>>>>>> imapc_user='someuser' -o imapc_password='secret' -D backup -n INBOX/ -R
>>>>>>> -u someuser at somedomain imapc: 2>&1 | tee /tmp/doveadm-backup.log
>>>>>>>
>>>>>>> Nov 03 09:06:35 dsync(someuser at somedomain): Warning: Mailbox changes
>>>>>>> caused a desync. You may want to run dsync again: Remote lost mailbox
>>>>>>> GUID c92f64f79f0d1ed01e6d5b314f04886c (maybe it was just deleted?)
>>>>>>>
>>>>>>> doveadm mailbox status -u someuser at somedomain all INBOX
>>>>>>> INBOX messages=0 recent=0 uidnext=1 uidvalidity=1577952633 unseen=0
>>>>>>> highestmodseq=1 vsize=0 guid=c92f64f79f0d1ed01e6d5b314f04886c
>>>>>>> firstsaved=never
>>>>>>>
>>>>>>> Any ideas what else to try or how to debug that further?
>>>>>>>
>>>>>>> I can send you the full log to your personal address, if that helps ...
>>>>>>>
>>>>>>> Ralf
>>>>>> You should rm -rf the target folder first. Can you attach `doveadm -D backup` logs? Check that it won't contain passwords.
>>>>> The mailbox directory did NOT exist before, therefore no need to rm -rf it.
>>>>>
>>>>> I send the logs to your private address only, I feel not comfortable to
>>>>> post them on the list.
>>>>>
>>>>> Ralf
>>>>>
>>>> 1. You did not delete the mailbox.
>>>>
>>>> 2. You are using **mbox** for subscription namespace, please don't.
>>>>
>>>> Also
>>>>
>>>> Nov 03 09:05:33 dsync(): Debug: brain M: Local mailbox tree: INBOX guid=c8adef115c84636335000000effb6190 uid_validity=1667466332 uid_next=1 subs=no last_change=0 last_subs=0
>>>> Nov 03 09:05:33 dsync(): Debug: brain S: Local mailbox tree: INBOX guid=c92f64f79f0d1ed01e6d5b314f04886c uid_validity=1577952633 uid_next=32746 subs=no last_change=0 last_subs=0
>>>>
>>>> This clearly shows that you did not, in fact, rm -rf the user's mailboxes prir running backup. Can you please try again, and clean up the user from t target mailsystem before running backup again?
>>>>
>>>> I would also strongly recommend not using ACL plugin while doing backup, unless you are backing up ACLs from source system.
>>> I removed all ACL plugin and config from my Dovecot config and removed
>>> the subscription namespace from my doveadm backup command, but now it
>>> fails with an error:
>>>
>>> doveadm -o imapc_user='someuser' -o imapc_password='secret' -D backup -n
>>> INBOX/ -R -u someuser at somedomain imapc:
>>>
>>> Nov 03 11:12:15 doveadm(someuser at somedomain 156): Debug: Effective
>>> uid=90, gid=101, home=/var/dovecot/imap/somedomain/someuser
>>> Nov 03 11:12:15 doveadm(someuser at somedomain 156): Debug: Home dir not
>>> found: /var/dovecot/imap/somedomain/someuser
>>>
>>> Nov 03 11:12:15 doveadm(someuser at somedomain): Error: namespace
>>> configuration error: subscriptions=yes namespace missing
>>>
>>> doveadm config -n is attached.
>>>
>>> Ralf
>> You can keep the subscription namespace in your config, otherwise you have no place to store subscriptions into. Just don't use mbox driver for it, use sdbox instead.
>>
>> namespace subscriptions {
>>    hidden = yes
>>    list = no
>>    location = sdbox:/index/directory/%n/subs-shared/:LISTINDEX=root:SUBSCRIPTIONS=subscriptions-shared:LAYOUT=INDEX
>>    prefix =
>>    separator = /
>>    subscriptions = yes
>> }
>>
>> Aki
> Oh and forgot to mention, for INBOX namespace you can leave subscriptions turned on. This is only needed for shared/public namespaces in any case..

I changed the config now, as you suggested (doveadm config -n is 
attached), unfortunately the result stays the same:

doveadm -o imapc_user='someuser' -o imapc_password='secret' -D backup -n 
INBOX/ -R -u someuser at somedomain imapc: 2>&1|tee 
/tmp/doveadm-backup-someuser2.log

Nov 03 11:35:39 dsync(someuser at somedomain): Warning: Mailbox changes 
caused a desync. You may want to run dsync again: Remote lost mailbox 
GUID c92f64f79f0d1ed01e6d5b314f04886c (maybe it was just deleted?)

doveadm mailbox status -u someuser at somedomain all INBOX
INBOX messages=0 recent=0 uidnext=1 uidvalidity=1577952633 unseen=0 
highestmodseq=1 vsize=0 guid=c92f64f79f0d1ed01e6d5b314f04886c 
firstsaved=never

I send the full log again to your private address.

Ralf

-- 
Ralf Becker
EGroupware GmbH [www.egroupware.org]
Handelsregister HRB Kaiserslautern 3587
Geschäftsführer Birgit und Ralf Becker
Leibnizstr. 17, 67663 Kaiserslautern, Germany
Telefon +49 631 31657-0
-------------- next part --------------
# 2.3.19.1 (9b53102964): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.5.19 (4eae2f79)
# OS: Linux 4.15.0-140-generic x86_64  
# Hostname: 6cc0d9aef9ad
auth_cache_negative_ttl = 2 mins
auth_cache_size = 10 M
auth_cache_ttl = 5 mins
auth_master_user_separator = *
auth_mechanisms = plain login
auth_username_chars = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_@#"
default_client_limit = 3500
default_process_limit = 512
disable_plaintext_auth = no
doveadm_port = 26
first_valid_uid = 90
imapc_features = rfc822.size fetch-headers
imapc_host = 10.44.88.3
imapc_list_prefix = INBOX
listen = *
log_path = /dev/stderr
login_greeting = Dovecot FRA.khs ready
mail_access_groups = dovecot
mail_attribute_dict = file:%h/dovecot-metadata
mail_fsync = never
mail_gid = dovecot
mail_location = mdbox:~/mdbox
mail_log_prefix = "%s(%u %p): "
mail_max_userip_connections = 200
mail_plugins = quota notify replication mail_log
mail_prefetch_count = 20
mail_uid = dovecot
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave vnd.dovecot.debug
mbox_min_index_size = 1000 B
mdbox_rotate_size = 50 M
namespace inboxes {
  inbox = yes
  location = 
  mailbox Drafts {
    special_use = \Drafts
  }
  mailbox Junk {
    special_use = \Junk
  }
  mailbox Sent {
    special_use = \Sent
  }
  mailbox Trash {
    special_use = \Trash
  }
  prefix = INBOX/
  separator = /
  subscriptions = yes
}
namespace subscriptions {
  hidden = yes
  list = no
  location = sdbox:/var/dovecot/index/%n/subs-shared/:LISTINDEX=root:SUBSCRIPTIONS=subscriptions-shared:LAYOUT=INDEX
  prefix = 
  separator = /
  subscriptions = yes
}
namespace users {
  location = mdbox:%%h/mdbox
  prefix = user/%%n/
  separator = /
  subscriptions = no
  type = shared
}
passdb {
  args = /etc/dovecot/dovecot-dict-master-auth.conf
  driver = dict
  master = yes
}
passdb {
  args = /etc/dovecot/dovecot-dict-auth.conf
  driver = dict
}
plugin {
  mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename
  mail_log_fields = uid box msgid size
  quota = dict:User quota::ns=INBOX/:file:%h/dovecot-quota
  quota_rule = *:storage=200GB
  sieve = ~/sieve/dovecot.sieve
  sieve_after = /var/dovecot/sieve/after.d/
  sieve_before = /var/dovecot/sieve/before.d/
  sieve_dir = ~/sieve
  sieve_extensions = +editheader
  sieve_user_log = ~/.sieve.log
}
postmaster_address = admins at egroupware.org
protocols = imap pop3 lmtp sieve
quota_full_tempfail = yes
service auth-worker {
  user = $default_internal_user
}
service auth {
  drop_priv_before_exec = no
  inet_listener {
    port = 113
  }
}
service doveadm {
  inet_listener {
    port = 26
  }
  vsz_limit = 640 M
}
service imap-login {
  inet_listener imap {
    port = 143
  }
  inet_listener imaps {
    port = 993
    ssl = yes
  }
  process_min_avail = 5
  service_count = 1
  vsz_limit = 64 M
}
service imap {
  executable = imap
  process_limit = 2048
  vsz_limit = 640 M
}
service lmtp {
  inet_listener lmtp {
    port = 24
  }
  unix_listener lmtp {
    mode = 0666
  }
  vsz_limit = 512 M
}
service managesieve-login {
  inet_listener sieve {
    port = 4190
  }
  inet_listener sieve_deprecated {
    port = 2000
  }
}
service pop3-login {
  inet_listener pop3 {
    port = 110
  }
  inet_listener pop3s {
    port = 995
    ssl = yes
  }
}
service pop3 {
  executable = pop3
}
service postlogin {
  executable = script-login -d rawlog -b -t
}
ssl_cert = </etc/certs/mail.egroupware.org.pem
ssl_cipher_list = ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
ssl_dh = # hidden, use -P to show it
ssl_key = # hidden, use -P to show it
userdb {
  args = /etc/dovecot/dovecot-sql.conf
  driver = sql
}
userdb {
  args = /etc/dovecot/dovecot-dict-auth.conf
  driver = dict
}
verbose_proctitle = yes
protocol lda {
  mail_plugins = quota notify replication mail_log sieve quota
}
protocol imap {
  imap_metadata = yes
  mail_max_userip_connections = 200
  mail_plugins = quota notify replication mail_log quota imap_quota
}
protocol lmtp {
  mail_max_lock_timeout = 25 secs
  mail_plugins = quota notify replication mail_log sieve quota
}


More information about the dovecot mailing list