Duplicate messages if message is moved when using dsync

[Thom Pol]

Hi,

Hope you are well.

We have a cluster of 2 Dovecot servers, both on v2.3.13 (89f716dc2), 
using dsync to sync the messages between them.

Previously, we used TCPS to sync the messages, but after some testing, 
we concluded that syncing over SSH resulted in a lot less failed syncs, 
so we started using SSH.

The change has been a success, but I now notice a issue when a email 
client immediately moves a messages to a separate folder while Dovecot 
is syncing, where the message is seen twice in the folder (with the 
exact same headers/content).

For example, I have set a filter in my email client, Thunderbird, to 
immediately move all emails coming from this list to a separate folder. 
When opening that folder, I do not see one, but two unread messages, 
both identical to each other.

When checking the directories on the server, I see this:
mx1:
/var/vmail/example.com/joe/Maildir/.Subdir.Subdir/cur/1652615808.M190190P990486.mx2,S=19089,W=19384:2,S
/var/vmail/example.com/joe/Maildir/.Subdir.Subdir/cur/1652615811.M180050P1376677.mx1,S=19089,W=19384:2,S

mx2:
/var/vmail/example.com/joe/Maildir/.Subdir.Subdir/cur/1652615811.M981426P990530.mx2,S=19089,W=19384:2,S
/var/vmail/example.com/joe/Maildir/.Subdir.Subdir/cur/1652615808.M190190P990486.mx2,S=19089,W=19384:2,S

Note the difference: on mx1, one indicates mx1, and one mx2, while on 
the other server, both indicate mx2/

Any idea (other then telling end-users not to use such filters) how we 
could prevent these duplicate messages?

This is our config:
# Pigeonhole version 0.5.13 (cdd19fe3)
# OS: Linux 5.10.0-13-cloud-amd64 x86_64 Debian 11.3
# Hostname: mx1.example.com
auth_mechanisms = plain login
disable_plaintext_auth = no
dsync_remote_cmd = ssh -p 222 -l%{login} %{host} doveadm dsync-server -u%u
imap_capability = +SPECIAL-USE XLIST
listen = *,[::]
lmtp_rcpt_check_quota = yes
log_timestamp = "%Y-%m-%d %H:%M:%S "
mail_max_userip_connections = 100
mail_plugins = quota
mail_privileged_group = vmail
namespace inbox {
   inbox = yes
   location =
   mailbox Drafts {
     special_use = \Drafts
   }
   mailbox Junk {
     special_use = \Junk
   }
   mailbox Sent {
     special_use = \Sent
   }
   mailbox "Sent Messages" {
     special_use = \Sent
   }
   mailbox Trash {
     special_use = \Trash
   }
   prefix =
   separator = .
}
passdb {
   args = /etc/dovecot/dovecot-sql.conf
   driver = sql
}
plugin {
   mail_replica = remote:root at mx2.example.com
   quota = dict:user::file:/var/vmail/%d/%n/.quotausage
   quota_status_nouser = DUNNO
   quota_status_overquota = 552 5.2.2 Mailbox is full
   quota_status_success = DUNNO
   sieve = /var/vmail/%d/%n/.sieve
   sieve_after = /var/vmail/%d/%n/.ispconfig.sieve
   sieve_before = /var/vmail/%d/%n/.ispconfig-before.sieve
   sieve_max_actions = 100
   sieve_max_redirects = 25
   sieve_max_script_size = 2M
}
protocols = imap pop3 lmtp
replication_max_conns = 50
service aggregator {
   fifo_listener replication-notify-fifo {
     mode = 0666
     user = vmail
   }
   unix_listener replication-notify {
     mode = 0666
     user = vmail
   }
}
service auth {
   unix_listener /var/spool/postfix/private/auth {
     group = postfix
     mode = 0660
     user = postfix
   }
   unix_listener auth-userdb {
     group = vmail
     mode = 0600
     user = vmail
   }
   user = root
}
service imap-login {
   client_limit = 1000
   process_limit = 512
}
service lmtp {
   unix_listener /var/spool/postfix/private/dovecot-lmtp {
     group = postfix
     mode = 0600
     user = postfix
   }
}
service quota-status {
   client_limit = 1
   executable = quota-status -p postfix
   unix_listener /var/spool/postfix/private/quota-status {
     group = postfix
     mode = 0660
     user = postfix
   }
}
service replicator {
   process_min_avail = 1
   unix_listener replicator-doveadm {
     mode = 0666
     user = vmail
   }
}
service stats {
   unix_listener stats-reader {
     group = vmail
     mode = 0660
     user = vmail
   }
   unix_listener stats-writer {
     group = vmail
     mode = 0660
     user = vmail
   }
}
ssl_cert = </etc/postfix/smtpd.cert
ssl_cipher_list = 
kEECDH:+kEECDH+SHA:kEDH:+kEDH+SHA:+kEDH+CAMELLIA:kECDH:+kECDH+SHA:kRSA:+kRSA+SHA:+kRSA+CAMELLIA:!aNULL:!eNULL:!SSLv2:!RC4:!MD5:!DES:!EXP:!SEED:!IDEA:!3DES
ssl_client_ca_dir = /etc/ssl/certs
ssl_dh = # hidden, use -P to show it
ssl_key = # hidden, use -P to show it
ssl_min_protocol = TLSv1.2
ssl_prefer_server_ciphers = yes
userdb {
   driver = prefetch
}
userdb {
   args = /etc/dovecot/dovecot-sql.conf
   driver = sql
}
protocol imap {
   auth_verbose = yes
   mail_plugins = quota quota imap_quota quota notify replication
}
protocol pop3 {
   auth_verbose = yes
   mail_plugins = quota quota quota notify replication
   pop3_uidl_format = %08Xu%08Xv
}
protocol lda {
   mail_plugins = quota sieve quota sieve quota notify replication
   postmaster_address = postmaster at mx1.example.com
}
protocol lmtp {
   mail_plugins = quota quota sieve sieve quota notify replication
   postmaster_address = postmaster at mx1.example.com
}

Thanks in advance for your thoughts.

Kind regards,

Thom Pol


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://dovecot.org/pipermail/dovecot/attachments/20220515/2cd51704/attachment.htm>