Replication weirdness
Daniel Miller
dmiller at amfes.com
Fri Dec 3 23:13:55 UTC 2021
And...
The user who has both a "Sent" and a "Sent Messages" now has:
drwx------ 3 vmail mail 24 Dec 3 09:56 Sent
lrwxrwxrwx 1 vmail mail 4 Nov 30 17:51 'Sent Messages' -> Sent
drwx------ 3 vmail mail 24 Dec 3 15:10 'Sent Messages-temp-1'
drwx------ 3 vmail mail 24 Dec 3 15:10 'Sent
Messages-temp-fc30bd0a3a9aaa61c1180500b88bfabe'
and I got the following errors:
Dec 3 15:10:46 cloud1 dovecot:
doveadm(obfuscated)<336247><EMEQBvWjqmF3IQUAuIv6vg>: Error: Duplicate
mailbox GUID 6aae8c39f3a3aa615a0700009db5accb for mailboxes Sent and
Sent Messages-temp-1 - giving a new GUID
63481f29f6a3aa6177210500b88bfabe to Sent
Dec 3 15:10:50 cloud1 dovecot:
doveadm(obfuscated)<336245><HrHeOPOjqmF1IQUAuIv6vg>: Panic: file
dsync-brain-mailbox.c: line 851 (dsync_brain_slave_recv_mailbox):
assertion failed: (memcmp(dsync_box->mailbox_guid,
local_dsync_box.mailbox_guid, sizeof(dsync_box->mailbox_guid)) == 0)
Dec 3 15:10:50 cloud1 dovecot:
doveadm(obfuscated)<336245><HrHeOPOjqmF1IQUAuIv6vg>: Error: Raw
backtrace: #0 fatal_handler_real[0x7fde7fd20060] -> #1
i_internal_fatal_handler[0x7fde7fd20190] -> #2 i_panic[0x7fde7fc731ff]
-> #3 dsync_brain_slave_recv_mailbox[0x55dde7b22900] -> #4
dsync_brain_run[0x55dde7b20380] -> #5 dsync_brain_run_io[0x55dde7b20b50]
-> #6 dsync_ibc_stream_input[0x55dde7b329c0] -> #7
io_loop_call_io[0x7fde7fd36500] -> #8
io_loop_handler_run_internal[0x7fde7fd37ac0] -> #9
io_loop_handler_run[0x7fde7fd365c0] -> #10 io_loop_run[0x7fde7fd36740]
-> #11 cmd_dsync_server_run[0x55dde7b04f60] -> #12
doveadm_mail_next_user[0x55dde7b06850] -> #13
doveadm_cmd_ver2_to_mail_cmd_wrapper[0x55dde7b077e0] -> #14
doveadm_cmd_run_ver2[0x55dde7b17f00] -> #15
client_connection_tcp_input[0x55dde7b1c6b0] -> #16
io_loop_call_io[0x7fde7fd36500] -> #17
io_loop_handler_run_internal[0x7fde7fd37ac0] -> #18
io_loop_handler_run[0x7fde7fd365c0] -> #19 io_loop_run[0x7fde7fd36740]
-> #20 master_service_run[0x7fde7fca87d0] -> #21 main[0x55dde7af7770] ->
#22 __libc_start_main[0x7fde7f8f9fc0] -> #23 _start[0x55dde7af78d0]
Dec 3 15:10:50 cloud1 dovecot:
doveadm(obfuscated)<336245><HrHeOPOjqmF1IQUAuIv6vg>: Fatal: master:
service(doveadm): child 336245 killed with signal 6 (core dumped)
Dec 3 15:10:52 cloud1 dovecot:
doveadm(obfuscated)<336253><2VTpM/ujqmF9IQUAuIv6vg>: Error: Duplicate
mailbox GUID 63481f29f6a3aa6177210500b88bfabe for mailboxes INBOX/Sent
and INBOX/Sent Messages-temp-1 - giving a new GUID
cba35507fca3aa617d210500b88bfabe to INBOX/Sent
Dec 3 15:10:58 cloud1 dovecot:
doveadm(obfuscated)<336258><ykbQAAGkqmGCIQUAuIv6vg>: Error: Duplicate
mailbox GUID dc3b4434fba3aa61660700009db5accb for mailboxes Sent and
Sent Messages-temp-1 - giving a new GUID
60ad190102a4aa6182210500b88bfabe to Sent
--
Daniel
------ Original Message ------
From: "Daniel Miller" <dmiller at amfes.com>
To: "Daniel Miller" <dmiller at amfes.com>; dovecot at dovecot.org
Sent: 12/3/2021 2:42:12 PM
Subject: Re[2]: Replication weirdness
>And...one more.
>
>I'm now seeing (again) messages like:
>
>Dec 3 14:29:14 cloud1 dovecot:
>doveadm(obfuscated)<334017><e3FHNjmaqmHBGAUAuIv6vg>: Error: Duplicate
>mailbox GUID bcb9ca36ae36aa617f0a00009db5accb for mailboxes INBOX/Sent
>Messages and INBOX/Sent - giving a new GUID
>fc30bd0a3a9aaa61c1180500b88bfabe to INBOX/Sent
>Dec 3 14:38:59 cloud1 dovecot:
>doveadm(obfuscated)<334394><an5KIoOcqmE6GgUAuIv6vg>: Error: Duplicate
>mailbox GUID fc30bd0a3a9aaa61c1180500b88bfabe for mailboxes INBOX/Sent
>Messages and INBOX/Sent - giving a new GUID
>f4338038839caa613a1a0500b88bfabe to INBOX/Sent
>
>Having one message for the initial sync I suppose is reasonable. A
>second...maybe? But I'm getting nervous I'm about to start seeing the
>endless temp folders again.
>--
>Daniel
>
>------ Original Message ------
>From: "Daniel Miller" <dmiller at amfes.com>
>To: "Daniel Miller" <dmiller at amfes.com>; dovecot at dovecot.org
>Sent: 12/3/2021 2:39:25 PM
>Subject: Re: Replication weirdness
>
>>Another item.
>>
>>Again, it may be a 2.3.13 issue and I'm now on 2.3.17. But...I had
>>problem when using the "-N" parameter for dsync. So - I just have
>>(had):
>>
>>replication_dsync_parameters = -d -l 30 -U -x INBOX/virtual -x
>>INBOX/shared
>>
>>Now that things are working - I wanted to have my other namespaces
>>sync as well. So I went to:
>>
>>replication_dsync_parameters = -d -l 30 -U -n INBOX -n INBOX/Archives
>>-n INBOX/Lists -x INBOX/virtual -x INBOX/shared
>>
>>This appears to be working (the sync is just starting)...but I'm
>>seeing lock errors in the logs such as:
>>Dec 3 14:34:24 bubba dovecot:
>>doveadm(dmiller at amfes.com)<31785><TV+0LlGbqmEpfAAAnbWsyw>: Error:
>>Couldn't lock /var/mail/amfes.com/dmiller/.dovecot-sync.lock:
>>fcntl(/var/mail/amfes.com/dmiller/.dovecot-sync.lock, write-lock,
>>F_SETLKW) locking failed: Timed out after 30 seconds (WRITE lock held
>>by pid 31373)
>>
>>Checking the pid in question I see it's actively syncing a folder in
>>my mailbox. So I'm guessing, purely guessing, that by having multiple
>>namespaces explicitly directed to sync Dovecot is trying to start a
>>sync process for each of those namespaces - but all of them share a
>>common lock and therefore only one operation is allowed at a time.
>>
>>Am I correct, and whether or not I am - how can I correct these
>>errors? Do I dare try going back to just "-N"?
>>
>>--
>>Daniel
>>
>>------ Original Message ------
>>From: "Daniel Miller" <dmiller at amfes.com>
>>To: dovecot at dovecot.org
>>Sent: 12/3/2021 2:16:28 PM
>>Subject: Replication weirdness
>>
>>>First, I have to say this. After configuring everything correctly -
>>>and that means *everything* correctly - Dovecot replication Just
>>>Works. I'm not sure how (yes I do - Timo & Co. Magic) - but it does.
>>>Real-time new sync is near instantaneous.
>>>
>>>Now the problem. Or the background for the problem. My primary server
>>>uses sdbox for primary storage, mdbox for archival storage, and
>>>fts-solr. I spun up a second server, using sdbox, mdbox, and
>>>fts-flatcurve. My namespaces are as defined below. As best I can tell
>>>(based on diff comparing two 'doveconf -n' outputs) my namespaces are
>>>the same on both servers.
>>>
>>>namespace archives {
>>> list = children
>>> location = mdbox:/var/mail/%d/%n/Archives/mdbox
>>> mailbox Unsorted {
>>> auto = no
>>> special_use = \Archive
>>> }
>>> prefix = INBOX/Archives/
>>> separator = /
>>> subscriptions = no
>>> type = private
>>>}
>>>namespace inbox {
>>> alias_for =
>>> hidden = no
>>> inbox = yes
>>> list = yes
>>> location =
>>> mailbox "Deleted Messages" {
>>> auto = no
>>> autoexpunge = 30 days
>>> special_use = \Trash
>>> }
>>> mailbox Drafts {
>>> auto = subscribe
>>> special_use = \Drafts
>>> }
>>> mailbox Sent {
>>> auto = subscribe
>>> special_use = \Sent
>>> }
>>> mailbox Trash {
>>> auto = subscribe
>>> autoexpunge = 30 days
>>> special_use = \Trash
>>> }
>>> prefix = INBOX/
>>> separator = /
>>> subscriptions = no
>>> type = private
>>>}
>>>namespace lists {
>>> list = children
>>> location = mdbox:/var/mail/%d/%n/Lists/mdbox
>>> prefix = INBOX/Lists/
>>> separator = /
>>> subscriptions = no
>>> type = private
>>>}
>>>namespace subscriptions {
>>> hidden = yes
>>> list = no
>>> location =
>>> prefix =
>>> separator = /
>>> subscriptions = yes
>>> type = private
>>>}
>>>namespace usershares {
>>> list = yes
>>> location = sdbox:/var/mail/%%d/%%n/sdbox:NO-NOSELECT
>>> prefix = INBOX/shared/%%d/%%n/
>>> separator = /
>>> subscriptions = no
>>> type = shared
>>>}
>>>namespace virtual {
>>> list = children
>>> location = virtual:/var/mail/%d/%n/virtual
>>> mailbox Flagged {
>>> comment = All my flagged messages
>>> special_use = \Flagged
>>> }
>>> prefix = INBOX/virtual/
>>> separator = /
>>> subscriptions = no
>>>}
>>>
>>>I also have:
>>>plugin {
>>> mailbox_alias_new = Sent Messages
>>> mailbox_alias_new2 = Sent Items
>>> mailbox_alias_new3 = Deleted Messages
>>> mailbox_alias_old = Sent
>>> mailbox_alias_old2 = Sent
>>> mailbox_alias_old3 = Trash
>>>}
>>>
>>>This setup worked fine with my single server. Then I enabled
>>>replication - just on the primary. Dsync went to work (it seemed to
>>>take forever for the initial sync but that's what happens with large
>>>mailboxes and slow internet connections).
>>>
>>>The problem came up with certain subfolders. And I believe it only
>>>happens with subfolders that have spaces in their names. I had two
>>>user's mailboxes (under Sent), one of which had a "Sent Messages"
>>>symlink alias for "Sent", that started generating tens or hundreds of
>>>duplicates during sync. Fortunately those subfolders only had a few
>>>mails in them. But I had trees looking like:
>>>
>>>[...] (below is under /var/mail/domain/user/sdbox/mailboxes/Sent/)
>>>Proposal
>>>Requests-temp-c6e003375e64a961c93d00009db5accb-temp-1-temp-f80b1a00ce9aa961a86-temp-2
>>>Proposal
>>>Requests-temp-c6e003375e64a961c93d00009db5accb-temp-1-temp-f80b1a00ce9aa961a86-temp-3
>>>Proposal
>>>Requests-temp-c6e003375e64a961c93d00009db5accb-temp-2-temp-023fa4271c9ca9611ade0400b88bfabe
>>>Proposal
>>>Requests-temp-c6e003375e64a961c93d00009db5accb-temp-2-temp-023fa4271c9ca9611ad-temp-1
>>>Proposal Requests-temp-c6e003375e64a961c93d00009db5accb-temp-2-temp-1
>>>Proposal Requests-temp-c6e003375e64a961c93d00009db5accb-temp-2-temp-2
>>>Proposal
>>>Requests-temp-c6e003375e64a961c93d00009db5accb-temp-2-temp-2-temp-1-temp-1
>>>Proposal Requests-temp-c6e003375e64a961c93d00009db5accb-temp-2-temp-3
>>>Proposal Requests-temp-c6e003375e64a961c93d00009db5accb-temp-2-temp-4
>>>Proposal Requests-temp-c6e003375e64a961c93d00009db5accb-temp-2-temp-5
>>>Proposal
>>>Requests-temp-c6e003375e64a961c93d00009db5accb-temp-2-temp-e2aa0f35c99ba961356500009db5accb
>>>[...]
>>>
>>>I kept stopping, cleaning up the folders, and re-starting - and they
>>>kept regenerating. I tried renaming the folders to eliminate the
>>>spaces and I think that helped in one case - for the others I just
>>>moved the folders outside of the mail area completely to let the sync
>>>finish.
>>>
>>>Now that it's been stable for a day or two - I enabled sync in the
>>>other direction. And after setting *all* the required parameters
>>>instead of just most of them...it's working. But...I'm nervous about
>>>moving the problem folders back over. I will say, if it makes any
>>>difference, my primary server *was* running version 2.3.13 and I just
>>>updated it to 2.3.17. The remote is also 2.3.17.
>>>
>>>--
>>>Daniel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://dovecot.org/pipermail/dovecot/attachments/20211203/4d9d85f1/attachment-0001.htm>
More information about the dovecot
mailing list