Duplicate messages if message is moved when using dsync
Paul Kudla (SCOM.CA Internet Services Inc.)
paul at scom.ca
Mon May 16 12:12:20 UTC 2022
ok duplicsate emails (even across dsync, replication etc) is typically
handled via a global sieve script
I use :
# cat duplicates.sieve
require "duplicate"; # for dovecot >= 2.2.18
if duplicate {
discard;
stop;
}
for the scripts
and setup sieve to work via my dovecot.conf file
relative parts below :
--------------------------------------------------------------------
protocols = imap pop3 lmtp sieve
protocol lmtp {
mail_plugins = $mail_plugins sieve
postmaster_address = monitor at scom.ca
}
protocol lda {
mail_plugins = $mail_plugins sieve
}
plugin {
.....
sieve = file:~/sieve;active=~/sieve/.dovecot.sieve
sieve_duplicate_default_period = 1h
sieve_duplicate_max_period = 1d
sieve_extensions = +duplicate +notify +imapflags +vacation-seconds
sieve_global_dir = /usr/local/etc/dovecot/sieve
sieve_before = /usr/local/etc/dovecot/sieve/duplicates.sieve
.....
}
service managesieve-login {
process_limit = 1000
vsz_limit = 1g
inet_listener sieve {
port = 4190
}
}
protocol sieve {
managesieve_implementation_string = Dovecot Pigeonhole
managesieve_max_line_length = 65536
}
------------------------------------------------------------------
note the sieve_before which handles duplictes during delivery etc.
Happy Monday !!!
Thanks - paul
Paul Kudla
Scom.ca Internet Services <http://www.scom.ca>
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3
Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266
On 5/15/2022 12:38 PM, Thom Pol wrote:
> Hi,
>
> Hope you are well.
>
> We have a cluster of 2 Dovecot servers, both on v2.3.13 (89f716dc2),
> using dsync to sync the messages between them.
>
> Previously, we used TCPS to sync the messages, but after some testing,
> we concluded that syncing over SSH resulted in a lot less failed syncs,
> so we started using SSH.
>
> The change has been a success, but I now notice a issue when a email
> client immediately moves a messages to a separate folder while Dovecot
> is syncing, where the message is seen twice in the folder (with the
> exact same headers/content).
>
> For example, I have set a filter in my email client, Thunderbird, to
> immediately move all emails coming from this list to a separate folder.
> When opening that folder, I do not see one, but two unread messages,
> both identical to each other.
>
> When checking the directories on the server, I see this:
> mx1:
> /var/vmail/example.com/joe/Maildir/.Subdir.Subdir/cur/1652615808.M190190P990486.mx2,S=19089,W=19384:2,S
> /var/vmail/example.com/joe/Maildir/.Subdir.Subdir/cur/1652615811.M180050P1376677.mx1,S=19089,W=19384:2,S
>
> mx2:
> /var/vmail/example.com/joe/Maildir/.Subdir.Subdir/cur/1652615811.M981426P990530.mx2,S=19089,W=19384:2,S
> /var/vmail/example.com/joe/Maildir/.Subdir.Subdir/cur/1652615808.M190190P990486.mx2,S=19089,W=19384:2,S
>
> Note the difference: on mx1, one indicates mx1, and one mx2, while on
> the other server, both indicate mx2/
>
> Any idea (other then telling end-users not to use such filters) how we
> could prevent these duplicate messages?
>
> This is our config:
> # Pigeonhole version 0.5.13 (cdd19fe3)
> # OS: Linux 5.10.0-13-cloud-amd64 x86_64 Debian 11.3
> # Hostname: mx1.example.com
> auth_mechanisms = plain login
> disable_plaintext_auth = no
> dsync_remote_cmd = ssh -p 222 -l%{login} %{host} doveadm dsync-server -u%u
> imap_capability = +SPECIAL-USE XLIST
> listen = *,[::]
> lmtp_rcpt_check_quota = yes
> log_timestamp = "%Y-%m-%d %H:%M:%S "
> mail_max_userip_connections = 100
> mail_plugins = quota
> mail_privileged_group = vmail
> namespace inbox {
> inbox = yes
> location =
> mailbox Drafts {
> special_use = \Drafts
> }
> mailbox Junk {
> special_use = \Junk
> }
> mailbox Sent {
> special_use = \Sent
> }
> mailbox "Sent Messages" {
> special_use = \Sent
> }
> mailbox Trash {
> special_use = \Trash
> }
> prefix =
> separator = .
> }
> passdb {
> args = /etc/dovecot/dovecot-sql.conf
> driver = sql
> }
> plugin {
> mail_replica = remote:root at mx2.example.com
> quota = dict:user::file:/var/vmail/%d/%n/.quotausage
> quota_status_nouser = DUNNO
> quota_status_overquota = 552 5.2.2 Mailbox is full
> quota_status_success = DUNNO
> sieve = /var/vmail/%d/%n/.sieve
> sieve_after = /var/vmail/%d/%n/.ispconfig.sieve
> sieve_before = /var/vmail/%d/%n/.ispconfig-before.sieve
> sieve_max_actions = 100
> sieve_max_redirects = 25
> sieve_max_script_size = 2M
> }
> protocols = imap pop3 lmtp
> replication_max_conns = 50
> service aggregator {
> fifo_listener replication-notify-fifo {
> mode = 0666
> user = vmail
> }
> unix_listener replication-notify {
> mode = 0666
> user = vmail
> }
> }
> service auth {
> unix_listener /var/spool/postfix/private/auth {
> group = postfix
> mode = 0660
> user = postfix
> }
> unix_listener auth-userdb {
> group = vmail
> mode = 0600
> user = vmail
> }
> user = root
> }
> service imap-login {
> client_limit = 1000
> process_limit = 512
> }
> service lmtp {
> unix_listener /var/spool/postfix/private/dovecot-lmtp {
> group = postfix
> mode = 0600
> user = postfix
> }
> }
> service quota-status {
> client_limit = 1
> executable = quota-status -p postfix
> unix_listener /var/spool/postfix/private/quota-status {
> group = postfix
> mode = 0660
> user = postfix
> }
> }
> service replicator {
> process_min_avail = 1
> unix_listener replicator-doveadm {
> mode = 0666
> user = vmail
> }
> }
> service stats {
> unix_listener stats-reader {
> group = vmail
> mode = 0660
> user = vmail
> }
> unix_listener stats-writer {
> group = vmail
> mode = 0660
> user = vmail
> }
> }
> ssl_cert = </etc/postfix/smtpd.cert
> ssl_cipher_list =
> kEECDH:+kEECDH+SHA:kEDH:+kEDH+SHA:+kEDH+CAMELLIA:kECDH:+kECDH+SHA:kRSA:+kRSA+SHA:+kRSA+CAMELLIA:!aNULL:!eNULL:!SSLv2:!RC4:!MD5:!DES:!EXP:!SEED:!IDEA:!3DES
> ssl_client_ca_dir = /etc/ssl/certs
> ssl_dh = # hidden, use -P to show it
> ssl_key = # hidden, use -P to show it
> ssl_min_protocol = TLSv1.2
> ssl_prefer_server_ciphers = yes
> userdb {
> driver = prefetch
> }
> userdb {
> args = /etc/dovecot/dovecot-sql.conf
> driver = sql
> }
> protocol imap {
> auth_verbose = yes
> mail_plugins = quota quota imap_quota quota notify replication
> }
> protocol pop3 {
> auth_verbose = yes
> mail_plugins = quota quota quota notify replication
> pop3_uidl_format = %08Xu%08Xv
> }
> protocol lda {
> mail_plugins = quota sieve quota sieve quota notify replication
> postmaster_address = postmaster at mx1.example.com
> }
> protocol lmtp {
> mail_plugins = quota quota sieve sieve quota notify replication
> postmaster_address = postmaster at mx1.example.com
> }
>
> Thanks in advance for your thoughts.
>
> Kind regards,
>
> Thom Pol
>
>
>
>
> --
> This message has been scanned for viruses and
> dangerous content by *MailScanner* <http://www.mailscanner.info/>, and is
> believed to be clean.
More information about the dovecot
mailing list