Replication weirdness
Daniel Miller
dmiller at amfes.com
Fri Dec 3 22:39:25 UTC 2021
Another item.
Again, it may be a 2.3.13 issue and I'm now on 2.3.17. But...I had
problem when using the "-N" parameter for dsync. So - I just have (had):
replication_dsync_parameters = -d -l 30 -U -x INBOX/virtual -x
INBOX/shared
Now that things are working - I wanted to have my other namespaces sync
as well. So I went to:
replication_dsync_parameters = -d -l 30 -U -n INBOX -n INBOX/Archives -n
INBOX/Lists -x INBOX/virtual -x INBOX/shared
This appears to be working (the sync is just starting)...but I'm seeing
lock errors in the logs such as:
Dec 3 14:34:24 bubba dovecot:
doveadm(dmiller at amfes.com)<31785><TV+0LlGbqmEpfAAAnbWsyw>: Error:
Couldn't lock /var/mail/amfes.com/dmiller/.dovecot-sync.lock:
fcntl(/var/mail/amfes.com/dmiller/.dovecot-sync.lock, write-lock,
F_SETLKW) locking failed: Timed out after 30 seconds (WRITE lock held by
pid 31373)
Checking the pid in question I see it's actively syncing a folder in my
mailbox. So I'm guessing, purely guessing, that by having multiple
namespaces explicitly directed to sync Dovecot is trying to start a sync
process for each of those namespaces - but all of them share a common
lock and therefore only one operation is allowed at a time.
Am I correct, and whether or not I am - how can I correct these errors?
Do I dare try going back to just "-N"?
--
Daniel
------ Original Message ------
From: "Daniel Miller" <dmiller at amfes.com>
To: dovecot at dovecot.org
Sent: 12/3/2021 2:16:28 PM
Subject: Replication weirdness
>First, I have to say this. After configuring everything correctly - and
>that means *everything* correctly - Dovecot replication Just Works. I'm
>not sure how (yes I do - Timo & Co. Magic) - but it does. Real-time new
>sync is near instantaneous.
>
>Now the problem. Or the background for the problem. My primary server
>uses sdbox for primary storage, mdbox for archival storage, and
>fts-solr. I spun up a second server, using sdbox, mdbox, and
>fts-flatcurve. My namespaces are as defined below. As best I can tell
>(based on diff comparing two 'doveconf -n' outputs) my namespaces are
>the same on both servers.
>
>namespace archives {
> list = children
> location = mdbox:/var/mail/%d/%n/Archives/mdbox
> mailbox Unsorted {
> auto = no
> special_use = \Archive
> }
> prefix = INBOX/Archives/
> separator = /
> subscriptions = no
> type = private
>}
>namespace inbox {
> alias_for =
> hidden = no
> inbox = yes
> list = yes
> location =
> mailbox "Deleted Messages" {
> auto = no
> autoexpunge = 30 days
> special_use = \Trash
> }
> mailbox Drafts {
> auto = subscribe
> special_use = \Drafts
> }
> mailbox Sent {
> auto = subscribe
> special_use = \Sent
> }
> mailbox Trash {
> auto = subscribe
> autoexpunge = 30 days
> special_use = \Trash
> }
> prefix = INBOX/
> separator = /
> subscriptions = no
> type = private
>}
>namespace lists {
> list = children
> location = mdbox:/var/mail/%d/%n/Lists/mdbox
> prefix = INBOX/Lists/
> separator = /
> subscriptions = no
> type = private
>}
>namespace subscriptions {
> hidden = yes
> list = no
> location =
> prefix =
> separator = /
> subscriptions = yes
> type = private
>}
>namespace usershares {
> list = yes
> location = sdbox:/var/mail/%%d/%%n/sdbox:NO-NOSELECT
> prefix = INBOX/shared/%%d/%%n/
> separator = /
> subscriptions = no
> type = shared
>}
>namespace virtual {
> list = children
> location = virtual:/var/mail/%d/%n/virtual
> mailbox Flagged {
> comment = All my flagged messages
> special_use = \Flagged
> }
> prefix = INBOX/virtual/
> separator = /
> subscriptions = no
>}
>
>I also have:
>plugin {
> mailbox_alias_new = Sent Messages
> mailbox_alias_new2 = Sent Items
> mailbox_alias_new3 = Deleted Messages
> mailbox_alias_old = Sent
> mailbox_alias_old2 = Sent
> mailbox_alias_old3 = Trash
>}
>
>This setup worked fine with my single server. Then I enabled
>replication - just on the primary. Dsync went to work (it seemed to
>take forever for the initial sync but that's what happens with large
>mailboxes and slow internet connections).
>
>The problem came up with certain subfolders. And I believe it only
>happens with subfolders that have spaces in their names. I had two
>user's mailboxes (under Sent), one of which had a "Sent Messages"
>symlink alias for "Sent", that started generating tens or hundreds of
>duplicates during sync. Fortunately those subfolders only had a few
>mails in them. But I had trees looking like:
>
>[...] (below is under /var/mail/domain/user/sdbox/mailboxes/Sent/)
>Proposal
>Requests-temp-c6e003375e64a961c93d00009db5accb-temp-1-temp-f80b1a00ce9aa961a86-temp-2
>Proposal
>Requests-temp-c6e003375e64a961c93d00009db5accb-temp-1-temp-f80b1a00ce9aa961a86-temp-3
>Proposal
>Requests-temp-c6e003375e64a961c93d00009db5accb-temp-2-temp-023fa4271c9ca9611ade0400b88bfabe
>Proposal
>Requests-temp-c6e003375e64a961c93d00009db5accb-temp-2-temp-023fa4271c9ca9611ad-temp-1
>Proposal Requests-temp-c6e003375e64a961c93d00009db5accb-temp-2-temp-1
>Proposal Requests-temp-c6e003375e64a961c93d00009db5accb-temp-2-temp-2
>Proposal
>Requests-temp-c6e003375e64a961c93d00009db5accb-temp-2-temp-2-temp-1-temp-1
>Proposal Requests-temp-c6e003375e64a961c93d00009db5accb-temp-2-temp-3
>Proposal Requests-temp-c6e003375e64a961c93d00009db5accb-temp-2-temp-4
>Proposal Requests-temp-c6e003375e64a961c93d00009db5accb-temp-2-temp-5
>Proposal
>Requests-temp-c6e003375e64a961c93d00009db5accb-temp-2-temp-e2aa0f35c99ba961356500009db5accb
>[...]
>
>I kept stopping, cleaning up the folders, and re-starting - and they
>kept regenerating. I tried renaming the folders to eliminate the spaces
>and I think that helped in one case - for the others I just moved the
>folders outside of the mail area completely to let the sync finish.
>
>Now that it's been stable for a day or two - I enabled sync in the
>other direction. And after setting *all* the required parameters
>instead of just most of them...it's working. But...I'm nervous about
>moving the problem folders back over. I will say, if it makes any
>difference, my primary server *was* running version 2.3.13 and I just
>updated it to 2.3.17. The remote is also 2.3.17.
>
>--
>Daniel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://dovecot.org/pipermail/dovecot/attachments/20211203/3f9fe476/attachment-0001.htm>
More information about the dovecot
mailing list