doveadm backup|sync works for every folder but INBOX
Ralf Becker
rb at egroupware.org
Thu Nov 3 11:43:48 UTC 2022
Hi Aki,
Am 03.11.22 um 12:27 schrieb Aki Tuomi:
>> On 03/11/2022 13:23 EET Aki Tuomi <aki.tuomi at open-xchange.com> wrote:
>>> On 03/11/2022 13:19 EET Ralf Becker <rb at egroupware.org> wrote:
>>>
>>>
>>> Hi Aki,
>>>
>>> Am 03.11.22 um 10:54 schrieb Aki Tuomi:
>>>>> On 03/11/2022 11:46 EET Ralf Becker <rb at egroupware.org> wrote:
>>>>>
>>>>>
>>>>> Hi Aki,
>>>>>
>>>>> Am 03.11.22 um 10:29 schrieb Aki Tuomi:
>>>>>>> On 03/11/2022 11:27 EET Ralf Becker <rb at egroupware.org> wrote:
>>>>>>>
>>>>>>>
>>>>>>> Hi Aki,
>>>>>>>
>>>>>>> Am 03.11.22 um 09:12 schrieb Aki Tuomi:
>>>>>>>>> On 03/11/2022 10:09 EET Ralf Becker <rb at egroupware.org> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hi Aki,
>>>>>>>>>
>>>>>>>>> Am 03.11.22 um 08:50 schrieb Aki Tuomi:
>>>>>>>>>>> On 03/11/2022 09:46 EET Ralf Becker <rb at egroupware.org> wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I'm trying to migrate an old Cyrus 2.5 server to Dovecot 2.3.19 using
>>>>>>>>>>> doveadm backup -R, which works for all folders but the INBOX itself,
>>>>>>>>>>> which always stays empty.
>>>>>>>>>>>
>>>>>>>>>>> The Cyrus side uses altnamespace:no and unixhierarchysep:no, it's used
>>>>>>>>>>> as imapc: remote in doveadm backup -R with imapc_list_prefix=INBOX
>>>>>>>>>>>
>>>>>>>>>>> Dovecot uses the following namespace to migrate into:
>>>>>>>>>>>
>>>>>>>>>>> namespace inboxes {
>>>>>>>>>>> inbox = yes
>>>>>>>>>>> location =
>>>>>>>>>>> mailbox Sent {
>>>>>>>>>>> auto = subscribe
>>>>>>>>>>> special_use = \Sent
>>>>>>>>>>> }
>>>>>>>>>>> ### some more folders omitted ###
>>>>>>>>>>> prefix = INBOX/
>>>>>>>>>>> separator = /
>>>>>>>>>>> subscriptions = no
>>>>>>>>>>> }
>>>>>>>>>> Hi!
>>>>>>>>>>
>>>>>>>>>> When syncing mailboxes from other server, you should use migration config file, which has **no** auto=subscribe or auto=create folders, as these can mess up with synchronization.
>>>>>>>>>>
>>>>>>>>>> Please see https://doc.dovecot.org/admin_manual/migrating_mailboxes/ for more details.
>>>>>>>>> Does a migration config file specified with doveadm -c <file> add to and
>>>>>>>>> overwrite the existing Dovecot configuration for the time the command
>>>>>>>>> runs, like the -o options, or do I need to start a separate server with
>>>>>>>>> a full configuration to e.g. have my authentication and mailbox location
>>>>>>>>> available?
>>>>>>>>>
>>>>>>>>> Ralf
>>>>>>>>>
>>>>>>>> It does not add/replace/overwrite configuration, you provide a fresh config file which is used *instead of* the default dovecot.conf.
>>>>>>>>
>>>>>>>> You don't need to run a separate instance necessarely, although in some larger migrations this has been used as well.
>>>>>>> I created now a separate instance with a modified configuration file
>>>>>>> with no auto=subscribe (or create), no replication and an empty storage.
>>>>>>> doveadm config -n is attached.
>>>>>>>
>>>>>>> Unfortunately the result is identical to my previous tries:
>>>>>>>
>>>>>>> doveadm -o namespace/subs/location=mbox:/var/dovecot/subs -o
>>>>>>> imapc_user='someuser' -o imapc_password='secret' -D backup -n INBOX/ -R
>>>>>>> -u someuser at somedomain imapc: 2>&1 | tee /tmp/doveadm-backup.log
>>>>>>>
>>>>>>> Nov 03 09:06:35 dsync(someuser at somedomain): Warning: Mailbox changes
>>>>>>> caused a desync. You may want to run dsync again: Remote lost mailbox
>>>>>>> GUID c92f64f79f0d1ed01e6d5b314f04886c (maybe it was just deleted?)
>>>>>>>
>>>>>>> doveadm mailbox status -u someuser at somedomain all INBOX
>>>>>>> INBOX messages=0 recent=0 uidnext=1 uidvalidity=1577952633 unseen=0
>>>>>>> highestmodseq=1 vsize=0 guid=c92f64f79f0d1ed01e6d5b314f04886c
>>>>>>> firstsaved=never
>>>>>>>
>>>>>>> Any ideas what else to try or how to debug that further?
>>>>>>>
>>>>>>> I can send you the full log to your personal address, if that helps ...
>>>>>>>
>>>>>>> Ralf
>>>>>> You should rm -rf the target folder first. Can you attach `doveadm -D backup` logs? Check that it won't contain passwords.
>>>>> The mailbox directory did NOT exist before, therefore no need to rm -rf it.
>>>>>
>>>>> I send the logs to your private address only, I feel not comfortable to
>>>>> post them on the list.
>>>>>
>>>>> Ralf
>>>>>
>>>> 1. You did not delete the mailbox.
>>>>
>>>> 2. You are using **mbox** for subscription namespace, please don't.
>>>>
>>>> Also
>>>>
>>>> Nov 03 09:05:33 dsync(): Debug: brain M: Local mailbox tree: INBOX guid=c8adef115c84636335000000effb6190 uid_validity=1667466332 uid_next=1 subs=no last_change=0 last_subs=0
>>>> Nov 03 09:05:33 dsync(): Debug: brain S: Local mailbox tree: INBOX guid=c92f64f79f0d1ed01e6d5b314f04886c uid_validity=1577952633 uid_next=32746 subs=no last_change=0 last_subs=0
>>>>
>>>> This clearly shows that you did not, in fact, rm -rf the user's mailboxes prir running backup. Can you please try again, and clean up the user from t target mailsystem before running backup again?
>>>>
>>>> I would also strongly recommend not using ACL plugin while doing backup, unless you are backing up ACLs from source system.
>>> I removed all ACL plugin and config from my Dovecot config and removed
>>> the subscription namespace from my doveadm backup command, but now it
>>> fails with an error:
>>>
>>> doveadm -o imapc_user='someuser' -o imapc_password='secret' -D backup -n
>>> INBOX/ -R -u someuser at somedomain imapc:
>>>
>>> Nov 03 11:12:15 doveadm(someuser at somedomain 156): Debug: Effective
>>> uid=90, gid=101, home=/var/dovecot/imap/somedomain/someuser
>>> Nov 03 11:12:15 doveadm(someuser at somedomain 156): Debug: Home dir not
>>> found: /var/dovecot/imap/somedomain/someuser
>>>
>>> Nov 03 11:12:15 doveadm(someuser at somedomain): Error: namespace
>>> configuration error: subscriptions=yes namespace missing
>>>
>>> doveadm config -n is attached.
>>>
>>> Ralf
>> You can keep the subscription namespace in your config, otherwise you have no place to store subscriptions into. Just don't use mbox driver for it, use sdbox instead.
>>
>> namespace subscriptions {
>> hidden = yes
>> list = no
>> location = sdbox:/index/directory/%n/subs-shared/:LISTINDEX=root:SUBSCRIPTIONS=subscriptions-shared:LAYOUT=INDEX
>> prefix =
>> separator = /
>> subscriptions = yes
>> }
>>
>> Aki
> Oh and forgot to mention, for INBOX namespace you can leave subscriptions turned on. This is only needed for shared/public namespaces in any case..
I changed the config now, as you suggested (doveadm config -n is
attached), unfortunately the result stays the same:
doveadm -o imapc_user='someuser' -o imapc_password='secret' -D backup -n
INBOX/ -R -u someuser at somedomain imapc: 2>&1|tee
/tmp/doveadm-backup-someuser2.log
Nov 03 11:35:39 dsync(someuser at somedomain): Warning: Mailbox changes
caused a desync. You may want to run dsync again: Remote lost mailbox
GUID c92f64f79f0d1ed01e6d5b314f04886c (maybe it was just deleted?)
doveadm mailbox status -u someuser at somedomain all INBOX
INBOX messages=0 recent=0 uidnext=1 uidvalidity=1577952633 unseen=0
highestmodseq=1 vsize=0 guid=c92f64f79f0d1ed01e6d5b314f04886c
firstsaved=never
I send the full log again to your private address.
Ralf
--
Ralf Becker
EGroupware GmbH [www.egroupware.org]
Handelsregister HRB Kaiserslautern 3587
Geschäftsführer Birgit und Ralf Becker
Leibnizstr. 17, 67663 Kaiserslautern, Germany
Telefon +49 631 31657-0
-------------- next part --------------
# 2.3.19.1 (9b53102964): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.5.19 (4eae2f79)
# OS: Linux 4.15.0-140-generic x86_64
# Hostname: 6cc0d9aef9ad
auth_cache_negative_ttl = 2 mins
auth_cache_size = 10 M
auth_cache_ttl = 5 mins
auth_master_user_separator = *
auth_mechanisms = plain login
auth_username_chars = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_@#"
default_client_limit = 3500
default_process_limit = 512
disable_plaintext_auth = no
doveadm_port = 26
first_valid_uid = 90
imapc_features = rfc822.size fetch-headers
imapc_host = 10.44.88.3
imapc_list_prefix = INBOX
listen = *
log_path = /dev/stderr
login_greeting = Dovecot FRA.khs ready
mail_access_groups = dovecot
mail_attribute_dict = file:%h/dovecot-metadata
mail_fsync = never
mail_gid = dovecot
mail_location = mdbox:~/mdbox
mail_log_prefix = "%s(%u %p): "
mail_max_userip_connections = 200
mail_plugins = quota notify replication mail_log
mail_prefetch_count = 20
mail_uid = dovecot
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave vnd.dovecot.debug
mbox_min_index_size = 1000 B
mdbox_rotate_size = 50 M
namespace inboxes {
inbox = yes
location =
mailbox Drafts {
special_use = \Drafts
}
mailbox Junk {
special_use = \Junk
}
mailbox Sent {
special_use = \Sent
}
mailbox Trash {
special_use = \Trash
}
prefix = INBOX/
separator = /
subscriptions = yes
}
namespace subscriptions {
hidden = yes
list = no
location = sdbox:/var/dovecot/index/%n/subs-shared/:LISTINDEX=root:SUBSCRIPTIONS=subscriptions-shared:LAYOUT=INDEX
prefix =
separator = /
subscriptions = yes
}
namespace users {
location = mdbox:%%h/mdbox
prefix = user/%%n/
separator = /
subscriptions = no
type = shared
}
passdb {
args = /etc/dovecot/dovecot-dict-master-auth.conf
driver = dict
master = yes
}
passdb {
args = /etc/dovecot/dovecot-dict-auth.conf
driver = dict
}
plugin {
mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename
mail_log_fields = uid box msgid size
quota = dict:User quota::ns=INBOX/:file:%h/dovecot-quota
quota_rule = *:storage=200GB
sieve = ~/sieve/dovecot.sieve
sieve_after = /var/dovecot/sieve/after.d/
sieve_before = /var/dovecot/sieve/before.d/
sieve_dir = ~/sieve
sieve_extensions = +editheader
sieve_user_log = ~/.sieve.log
}
postmaster_address = admins at egroupware.org
protocols = imap pop3 lmtp sieve
quota_full_tempfail = yes
service auth-worker {
user = $default_internal_user
}
service auth {
drop_priv_before_exec = no
inet_listener {
port = 113
}
}
service doveadm {
inet_listener {
port = 26
}
vsz_limit = 640 M
}
service imap-login {
inet_listener imap {
port = 143
}
inet_listener imaps {
port = 993
ssl = yes
}
process_min_avail = 5
service_count = 1
vsz_limit = 64 M
}
service imap {
executable = imap
process_limit = 2048
vsz_limit = 640 M
}
service lmtp {
inet_listener lmtp {
port = 24
}
unix_listener lmtp {
mode = 0666
}
vsz_limit = 512 M
}
service managesieve-login {
inet_listener sieve {
port = 4190
}
inet_listener sieve_deprecated {
port = 2000
}
}
service pop3-login {
inet_listener pop3 {
port = 110
}
inet_listener pop3s {
port = 995
ssl = yes
}
}
service pop3 {
executable = pop3
}
service postlogin {
executable = script-login -d rawlog -b -t
}
ssl_cert = </etc/certs/mail.egroupware.org.pem
ssl_cipher_list = ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
ssl_dh = # hidden, use -P to show it
ssl_key = # hidden, use -P to show it
userdb {
args = /etc/dovecot/dovecot-sql.conf
driver = sql
}
userdb {
args = /etc/dovecot/dovecot-dict-auth.conf
driver = dict
}
verbose_proctitle = yes
protocol lda {
mail_plugins = quota notify replication mail_log sieve quota
}
protocol imap {
imap_metadata = yes
mail_max_userip_connections = 200
mail_plugins = quota notify replication mail_log quota imap_quota
}
protocol lmtp {
mail_max_lock_timeout = 25 secs
mail_plugins = quota notify replication mail_log sieve quota
}
More information about the dovecot
mailing list