[Dovecot] dsync deleted my mailbox - what did I do wrong?
Hi.
Mostly annoying: I migrated from one machine to another, made sure the target host worked as expected, updated mx records and - after a couple of days - signed it off as good. This is just my private machine, no big deal if something goes wrong..
Everything's fine? Good, let's migrate my inbox from the old machine. There's no direct connectivity between those servers, so what I did was:
(old server) sudo -u vmail dsync -u dar@darklajid.de backup maildir:/tmp/mail_backup/
Works fine, got my maildir. Tar'd it up, moved it to the new server. Now how do I import those mails? Ah, let's use the same command, with -R?
(new server) sudo -u vmail dsync -u dar@darklajid.de -R backup maildir:/tmp/mail_backup Error: Mailbox INBOX sync: mailbox_delete failed: INBOX can't be deleted.
Wait. What? Sure enough, the last couple of days are gone, the target mailbox is completely empty. I read the man page over and over again, but failed to see the problem. I even thought for a moment that _maybe_ dsync backup is one-way only (even if that failed as well) and tried the same command with mirror. Exactly the same output, same result, empty target mailbox.
In the end I succeeded to import the mails with doveadm import, completely lost a number of days of mails. My fault, sloppy not to back the up again, but I still don't think that this should happen. Ever.
My question now is:
where did I fail (ignoring the backups, please. That's .. something I know)
Can I use dsync ... for backups? I don't think that this is a good idea after yesterday night?
Should dsync EVER try to delete mailboxes? Even 'special' mailboxes? Should it warn about that, asking for a --force switch or something?
Any insights would be appreciated. At this point the damage is done, but I'd like to learn how to do better.
Ben
Hi
- where did I fail (ignoring the backups, please. That's .. something I know)
From the man page: "backup - Backup mails from default mail location to location2 (or vice versa, if -R parameter is given). No changes are ever done to the source location. Any changes done in destination are discarded."
The last sentence describes what happened to you: all new mail on the new machine is a "change" and was discarded (by deleting new mail.) If I'm not mistaken, this is correct behaviour for backup mode - you get exact copy of the source side (maildir:/tmp/mail_backup) on destination side (dar@darklajid.de)
- Can I use dsync ... for backups? I don't think that this is a good idea after yesterday night?
AFAIK you can safely use it to make the backup. I'm not sure if it can be reliably used to restore data (don't think so but I'm not an expert.) I'd use doveadm import for that.
Hey Jiri.
Thanks for getting back.
On Friday, April 4, 2014 4:48:48 PM CEST, Jiri Bourek wrote:
- where did I fail (ignoring the backups, please. That's .. something I know)
From the man page: "backup - Backup mails from default mail location to location2 (or vice versa, if -R parameter is given). No changes are ever done to the source location. Any changes done in destination are discarded."
Yeah, maybe. That's what I thought _after_ the fact (i.e. that was what I hinted at with 'one way sync'). But see below.
The last sentence describes what happened to you: all new mail on the new machine is a "change" and was discarded (by deleting new mail.) If I'm not mistaken, this is correct behaviour for backup mode - you get exact copy of the source side (maildir:/tmp/mail_backup) on destination side (dar@darklajid.de)
That would be sort of okay. Except that isn't what happened:
- The target mailbox was killed completely
- Nothing was restored
If what you're suggesting here is true I'd expect a clean copy of my source
- even if it destroys all other changes. That did NOT happen though. It nuked the target and didn't restore a thing.
Plus, dsync mirror did exactly the same: Nuked the (live) mailbox once more, same error message, not a single message restored (but also no modification to the source).
- Can I use dsync ... for backups? I don't think that this is a good idea after yesterday night?
AFAIK you can safely use it to make the backup. I'm not sure if it can be reliably used to restore data (don't think so but I'm not an expert.) I'd use doveadm import for that.
That'd be my experience at this point as well, of course. :-) The bigger question is if this is well-known / correct and if this should be documented in a better fashion. Was I really that naive to expect that to work (in that case: ignore the documentation request) or could that happen again?
Ben
The last sentence describes what happened to you: all new mail on the new machine is a "change" and was discarded (by deleting new mail.) If I'm not mistaken, this is correct behaviour for backup mode - you get exact copy of the source side (maildir:/tmp/mail_backup) on destination side (dar@darklajid.de)
That would be sort of okay. Except that isn't what happened:
- The target mailbox was killed completely
- Nothing was restored
If what you're suggesting here is true I'd expect a clean copy of my source - even if it destroys all other changes. That did NOT happen though. It nuked the target and didn't restore a thing.
True - if we move from "problem is dsync deleted new mail" to "problem is dsync was unable to restore the backup", the described behaviour looks like a bug to me too. It may have something to do with the maildir format, I recall some discussion regarding folder INBOX, which needs special handling (because it's physically stored in maildir root, whereas every other folder is stored in folder-named subdirectory)
That said, I tried something along what you did and it failed for me too. So I deleted the mailbox completely, recreated it, tried again and this time the restore succeeded. It seems the easiest possible way to reproduce the faulty behaviour is:
- create mailbox for testing, here test@example.com
- create IMAP folder under INBOX ( namespace inbox { separator = / } )
doveadm mailbox create -u test@example.com INBOX/test
- attempt to restore from backup
doveadm backup -u test@example.com -R maildir:/mnt/mail-backups/test/
which yields (on Dovecot 2.2.12)
dsync(test@example.com): Error: Mailbox INBOX sync: mailbox_delete failed: INBOX can't be deleted.
Another try shows that IMAP folder created somewhere else (not under INBOX) isn't a problem:
# doveadm mailbox create -u test@example.com testtest # doveadm backup -u test@example.com -R maildir:/mnt/mail-backups/test/
This yields no output, folder testtest is deleted (as expected), INBOX is populated from backup.
Another try, this time I used mbox instead of maildir by specifying
-o mail_location=mbox:/path/test/mail
to doveadm. Worked without error even with INBOX/test folder (which got deleted during restore)
No idea if this can be considered as a bug, or the test does something that is not supposed to be done in the first place (Although different results with different storage format suggests a bug to me.)
Plus, dsync mirror did exactly the same: Nuked the (live) mailbox once more, same error message, not a single message restored (but also no modification to the source).
I was doing some trial and error testing with doveadm sync (should be the same as dsync mirror.) If used on a mailbox which has seen some changes, this sync's behaviour is just strange.
Or - to be more precise - it seems strange on first sight, but when you think about it, it does what is supposed to do. The sync mode is (AFAIK) designed to keep single mailbox synchronized on two hosts. If you created new mailbox on the new host, then had some mail delivered to it and after some time decided to add mail from old host, then you don't have single mailbox - you have two mailboxes with the same name.
In other words this scenario is probably something dsync wasn't designed to be used for and there's no surprise mirror mode can't handle it.
And again - I'm no expert, so it's entirely possible everything I wrote here is complete and utter nonsense Let's hope someone more knowledgeable corrects me if that is the case.
On 7/4/2014 4:01 μμ, Jiri Bourek wrote:
I was doing some trial and error testing with doveadm sync (should be the same as dsync mirror.) If used on a mailbox which has seen some changes, this sync's behaviour is just strange.
I can confirm (on 2.2.12) that the behavior is the same using replication (mirroring). Creating a folder on either end, e.g. as a subfolder of Inbox, does not create an identical new folder on the other end. I would expect that the folder is replicated!
Manual dsync from the CLI will not replicate folders as well.
I will agree with Jiri that is a strange (wrong?) behavior in dsync. Folders should be replicated, whether new or not. If not, how message moves between older and newer folders can be replicated on the fly?
Please advise on how to handle this situation because if folders are created/deleted/moved by users, dsync may lead to a chaos!
Nick
Hello
Many different dsync issues have come up in this thread. Ill try to answer them as best as I can.
dsync backup -R The conclusion reached in the thread was correct. Instead of the backup option, doveadm import would be better suited for merging old mails into an existing mailbox.
Maildir + INBOX + backup/sync/replicate In the test scenarios where the INBOX on one side was to be completely removed, e.g. doveadm backup -R the dsync failed and nothing was synced to the target. This is because before moving the source mails to the mailbox, dsync cleans out the old ones ( -R preserves nothing) and in Maildir the INBOX can not be removed. This is a feature/not easily solvable, because in Maildir INBOX is different from other folders.
dsync replication / doveadm sync not working as expected. These came in pretty late in the thread and I did not get a full picture of what kind of setups and parameters were used. I suspect these might be a configuration issue. I think trying with different configurations and going through the documentation, such as it is, once more, is your best bet. Use -D and -v to make dsync more verbose, so you do not miss any error messages.
br, Teemu Huovila
On 8/4/2014 12:38 μμ, Teemu Huovila wrote:
- dsync replication / doveadm sync not working as expected. These came in pretty late in the thread and I did not get a full picture of what kind of setups and parameters were used. I suspect these might be a configuration issue. I think trying with different configurations and going through the documentation, such as it is, once more, is your best bet. Use -D and -v to make dsync more verbose, so you do not miss any error messages.
Thank you for the reply; I am focusing on the 3rd part, since this is the one I can provide feedback about.
My current configuration is exactly as suggested on the wiki2 and I list it below for your reference.
Neither using replication nor using dsync from CLI leads to subfolders getting replicated, as I have explained. As an example, if a user creates subfolder "boxtest" e.g. under Inbox on either side, it never gets created on the other side.
Running dsync with -Dv does not reveal any errors.
For example, here is the output of command:
# dsync -fDv -u imaptester mirror ssh -l root vmail1.example.com dsync -u imaptester
Mailbox "boxtest" under Inbox (on vmail server) -containing one message- should get replicated (created) on vmail1 server, but it does not.
If you can spot anything that can help on tracing the problem, please help.
Otherwise, I can't see what is causing the erratic replication.
Thanks, Nick
... dsync-local(imaptester): Debug: brain M: in box 'INBOX.boxtest' recv_state=mailbox send_state=mailbox dsync-local(imaptester): Debug: brain M: out box 'INBOX.boxtest' recv_state=mailbox send_state=mailbox changed=0 dsync-local(imaptester): Debug: brain M: out state=sync_mails changed=0 dsync-local(imaptester): Debug: brain M: in state=sync_mails dsync-local(imaptester): Debug: brain M: in box 'INBOX.boxtest' recv_state=mailbox send_state=mailbox dsync-local(imaptester): Debug: brain M: out box 'INBOX.boxtest' recv_state=attributes send_state=changes changed=1 dsync-local(imaptester): Debug: brain M: out state=sync_mails changed=1 dsync-local(imaptester): Debug: brain M: in state=sync_mails dsync-local(imaptester): Debug: brain M: in box 'INBOX.boxtest' recv_state=attributes send_state=changes dsync-local(imaptester): Debug: brain M: out box 'INBOX.boxtest' recv_state=changes send_state=mail_requests changed=1 dsync-local(imaptester): Debug: brain M: out state=sync_mails changed=1 dsync-local(imaptester): Debug: brain M: in state=sync_mails dsync-local(imaptester): Debug: brain M: in box 'INBOX.boxtest' recv_state=changes send_state=mail_requests dsync-local(imaptester): Debug: brain M: Import INBOX.boxtest: Import change GUID=1396119018.M550517P3113.vmail.example.com,S=1169,W=1194 UID=1 hdr_hash= dsync-local(imaptester): Debug: brain M: out box 'INBOX.boxtest' recv_state=changes send_state=mail_requests changed=1 dsync-local(imaptester): Debug: brain M: out state=sync_mails changed=1 dsync-local(imaptester): Debug: brain M: in state=sync_mails dsync-local(imaptester): Debug: brain M: in box 'INBOX.boxtest' recv_state=changes send_state=mail_requests dsync-local(imaptester): Debug: brain M: Import INBOX.boxtest: Last common UID=1 dsync-local(imaptester): Debug: brain M: out box 'INBOX.boxtest' recv_state=mail_requests send_state=mails changed=1 dsync-local(imaptester): Debug: brain M: out state=sync_mails changed=1 dsync-local(imaptester): Debug: brain M: in state=sync_mails dsync-local(imaptester): Debug: brain M: in box 'INBOX.boxtest' recv_state=mail_requests send_state=mails dsync-local(imaptester): Debug: brain M: out box 'INBOX.boxtest' recv_state=mail_requests send_state=mails changed=0 dsync-local(imaptester): Debug: brain M: out state=sync_mails changed=0 dsync-local(imaptester): Debug: brain M: in state=sync_mails dsync-local(imaptester): Debug: brain M: in box 'INBOX.boxtest' recv_state=mail_requests send_state=mails dsync-local(imaptester): Debug: brain M: out box 'INBOX.boxtest' recv_state=mails send_state=done changed=1 dsync-local(imaptester): Debug: brain M: out state=sync_mails changed=1 dsync-local(imaptester): Debug: brain M: in state=sync_mails dsync-local(imaptester): Debug: brain M: in box 'INBOX.boxtest' recv_state=mails send_state=done dsync-local(imaptester): Debug: brain M: Import INBOX.boxtest: Saved UIDs: dsync-local(imaptester): Debug: brain M: Import INBOX.boxtest: Finish update: min_next_uid=2 min_first_recent_uid=2 min_highest_modseq=2 min_highest_pvt_modseq=0 dsync-local(imaptester): Debug: brain M: out box 'INBOX.boxtest' recv_state=recv_last_common send_state=done changed=1 dsync-local(imaptester): Debug: brain M: out state=sync_mails changed=1 dsync-local(imaptester): Debug: brain M: in state=sync_mails dsync-local(imaptester): Debug: brain M: in box 'INBOX.boxtest' recv_state=recv_last_common send_state=done dsync-local(imaptester): Debug: brain M: out box 'INBOX.boxtest' recv_state=recv_last_common send_state=done changed=0 dsync-local(imaptester): Debug: brain M: out state=sync_mails changed=0 dsync-local(imaptester): Debug: brain M: in state=sync_mails dsync-local(imaptester): Debug: brain M: in box 'INBOX.boxtest' recv_state=recv_last_common send_state=done dsync-local(imaptester): Debug: brain M: out box '' recv_state=recv_last_common send_state=done changed=1 dsync-local(imaptester): Debug: brain M: out state=master_send_mailbox changed=1 dsync-local(imaptester): Debug: brain M: in state=master_send_mailbox dsync-local(imaptester): Debug: brain M: out state=sync_mails changed=1 dsync-local(imaptester): Debug: brain M: in state=sync_mails ...
============================ Configuration file
# cat /etc/dovecot/dovecot.conf # 2.2.12: dovecot.conf
protocols = imap pop3
login_greeting = EXAMPLE.COM POP/IMAP Srv XAPITI XPICTOY
mail_location = maildir:~/Maildir/ mail_gid = 500 mail_uid = 500
auth_mechanisms = plain login auth_username_format = %Lu
auth_verbose = yes auth_debug = no
disable_plaintext_auth = no
mail_plugins = quota notify replication
protocol imap { imap_client_workarounds = "delay-newmail" mail_plugins = quota imap_quota notify replication }
protocol pop3 { mail_max_userip_connections = 3 mail_plugins = quota notify replication pop3_client_workarounds = outlook-no-nuls oe-ns-eoh pop3_uidl_format = %08Xu%08Xv }
protocol lda { auth_socket_path = /var/run/dovecot/auth-master info_log_path = log_path = mail_plugins = quota notify replication postmaster_address = sysadmin@example.com sendmail_path = /usr/lib/sendmail }
userdb { args = /etc/dovecot/dovecot-usrdb-ldap.conf driver = ldap }
passdb { args = /etc/dovecot/dovecot-passdb-ldap.conf driver = ldap }
dsync_remote_cmd = ssh -l root vmail1.example.com doveadm dsync-server -u%u
replication_dsync_parameters = -d -N -l 30 -U
plugin { mail_replica = remote:vmail@vmail1.example.com }
plugin { quota = maildir:User quota quota_rule = *:storage=4G quota_rule2 = Trash:storage=+3%% quota_warning = storage=75%% quota-warning 75 %u quota_warning2 = storage=90%% quota-warning 90 %u }
service quota-warning { executable = script /opt/mail1.sh user = vmail unix_listener quota-warning { user = vmail } }
service aggregator { fifo_listener replication-notify-fifo { user = vmail } unix_listener replication-notify { user = vmail } }
service replicator { unix_listener replicator-doveadm { mode = 0600 } }
service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0660 user = postfix } unix_listener auth-master { group = vmail mode = 0660 user = vmail } user = root }
service imap-login { service_count = 1 vsz_limit = 64 M }
service pop3-login { service_count = 1 vsz_limit = 64 M }
service replicator { process_min_avail = 1 }
ssl_ca =
syslog_facility = local1
================================================================================
On 04/08/2014 03:00 PM, Nikolaos Milas wrote:
Neither using replication nor using dsync from CLI leads to subfolders getting replicated, as I have explained. As an example, if a user creates subfolder "boxtest" e.g. under Inbox on either side, it never gets created on the other side. I cant find any errors, but I might be missing something obvious. I only have a few suggestions for things to check.
You listed the config for one host (vmail i assume). Is the configuration similar on the vmail1 side? Especially, can the command dsync -u <user> find the correct location for the users mails?
For the replicator plugin scenario, does doveadm have access to auth, i.e. does doveadm user '*' work on both sides?
Are the dovecot instances running on different hosts (dovecot --hostdomain is different)?
Instead of "dsync mirror", try using the v2.2 syntax "doveadm sync". Also, i _think_ you need to execute dsync-server on the other side, so your full command becomes: doveadm sync -u imaptester ssh -l root vmail1.example.com doveadm dsync-server -u imaptester Sadly, there is no man-page for doveadm sync yet.
br, Teemu Huovila
On 8/4/2014 4:47 μμ, Teemu Huovila wrote:
I cant find any errors, but I might be missing something obvious. I only have a few suggestions for things to check.
- You listed the config for one host (vmail i assume). Is the configuration similar on the vmail1 side? Especially, can the command dsync -u <user> find the correct location for the users mails?
Yes, mail is replicated in existing folders, but new subfolders don't get replicated.
The config I listed was on the vmail (master) side, indeed. Config on the vmail1 (replicated) side is identical except the replication parts. I list it at the end of this mail for your reference.
- For the replicator plugin scenario, does doveadm have access to auth, i.e. does doveadm user '*' work on both sides?
Yes. Everything is fine: 317 identical accounts on each side (read from replicated LDAP).
- Are the dovecot instances running on different hosts (dovecot --hostdomain is different)?
Yes: vmail.example.com vs vmail1.example.com
- Instead of "dsync mirror", try using the v2.2 syntax "doveadm sync". Also, i_think_ you need to execute dsync-server on the other side, so your full command becomes: doveadm sync -u imaptester ssh -l root vmail1.example.com doveadm dsync-server -u imaptester Sadly, there is no man-page for doveadm sync yet.
OK, I ran the command:
doveadm -Dv sync -u imaptester ssh -l root vmail1.example.com doveadm dsync-server -u imaptester
The output is similar. Still no subfolder sync (but individual messages on existing subfolders get sync'ed):
... dsync-local(imaptester): Debug: brain M: in box 'INBOX.boxtest' recv_state=mailbox send_state=mailbox dsync-local(imaptester): Debug: brain M: out box 'INBOX.boxtest' recv_state=mailbox send_state=mailbox changed=0 dsync-local(imaptester): Debug: brain M: out state=sync_mails changed=0 dsync-local(imaptester): Debug: brain M: in state=sync_mails dsync-local(imaptester): Debug: brain M: in box 'INBOX.boxtest' recv_state=mailbox send_state=mailbox dsync-local(imaptester): Debug: brain M: out box '' recv_state=mailbox send_state=mailbox changed=1 dsync-local(imaptester): Debug: brain M: out state=master_send_mailbox changed=1 dsync-local(imaptester): Debug: brain M: in state=master_send_mailbox dsync-local(imaptester): Debug: brain M: out state=sync_mails changed=1 dsync-local(imaptester): Debug: brain M: in state=sync_mails ...
Please suggest any other ideas!
Thanks, Nick
================================== vmail1 Config file
# cat /etc/dovecot/dovecot.conf # 2.2.12: dovecot.conf
protocols = imap pop3
login_greeting = VMAIL1 POP/IMAP Srv XAPITI XPICTOY
mail_location = maildir:~/Maildir/ mail_gid = 5000 mail_uid = 5000
auth_mechanisms = plain login auth_username_format = %Lu auth_verbose = yes disable_plaintext_auth = no
mail_plugins = quota
protocol imap { imap_client_workarounds = "delay-newmail " mail_plugins = quota imap_quota }
protocol pop3 { mail_max_userip_connections = 3 mail_plugins = quota pop3_client_workarounds = outlook-no-nuls oe-ns-eoh pop3_uidl_format = %08Xu%08Xv }
protocol lda { auth_socket_path = /var/run/dovecot/auth-master info_log_path = log_path = mail_plugins = quota postmaster_address = sysadmin@example.com sendmail_path = /usr/lib/sendmail }
userdb { args = /etc/dovecot/dovecot-usrdb-ldap.conf driver = ldap }
passdb { args = /etc/dovecot/dovecot-passdb-ldap.conf driver = ldap }
plugin { quota = maildir:User quota quota_rule = *:storage=4G quota_rule2 = Trash:storage=+3%% quota_warning = storage=75%% quota-warning 75 %u quota_warning2 = storage=90%% quota-warning 90 %u }
service quota-warning { executable = script /opt/mail1.sh user = vmail unix_listener quota-warning { user = vmail } }
service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0660 user = postfix } unix_listener auth-master { group = vmail mode = 0660 user = vmail } user = root }
service imap-login { service_count = 1 vsz_limit = 64 M }
service pop3-login { service_count = 1 vsz_limit = 64 M }
ssl_ca =
syslog_facility = local1
========================================================================================
On 8/4/2014 5:20 μμ, Nikolaos Milas wrote:
Still no subfolder sync (but individual messages on existing subfolders get sync'ed):
Given that I can't make dsync work properly, until this thread provides info that might lead to correct operation, which I hope, I am thinking of trying syncing using lsyncd & unison (instead of dsync).
Before moving on, I would like to kindly ask list members:
Are there any admins around who have setup dsync replication and see new subfolder replication work correctly?
Is lsyncd & unison expected to work correctly given the particular architecture of Dovecot? (I would think it should work.) Anyone has tried it?
Thanks, Nick
participants (4)
-
Benjamin Podszun
-
Jiri Bourek
-
Nikolaos Milas
-
Teemu Huovila