Question regarding replication - duplicate emails

Fri Oct 20 20:01:05 EEST 2017

Hi,

Anyone has suggestions? The situation also happends when I “delete” a message from my Phone.
It seems to get replicated instantly and the just deleted email is back in the mailbox again.

I did remove HA Proxy support in the meantime to rule that out and I have enabled the default
replication_max_conns.

thanks
Remko

> On 13 Oct 2017, at 11:56, Remko Lodder <remko at FreeBSD.org> wrote:
> 
> Dear Dovecot and community,
> 
> We run a small email service for our customers, based on two machines that are made “redundant or clustered” by using the replication feature of Dovecot.
> This works well, for most emails.
> 
> Sometimes the following happends:
> 
> Email to our support database arrives at the inbox.
> Every period a cronjob looks into that mailbox and parses the information and makes a support ticket from that message.
> The cronjob deletes the email afterwards and sees whether there are more new mails or not.
> 
> In between the deletion and the (next) check, the email gets “resynced” again from the “other machine”, and is seen as a new
> email by the cronjob and gets handled again. So basically we get two tickets for one email. After that second run the email is
> no longer replicated.
> 
> This behaviour can also be seen via webmail and email clients, this suggests that the replication might not be aware (enough) that
> the replication is occurring or that an item already had been synced from A to B (and does not need to get back from B to A after A
> deleted it).
> 
> Is there a way to fiddle with the acknowledgement timing or give the processes some more space/time to get on par with eachother?
> 
> Below is the configuration of machine B, they are synchronised through puppet, so only the hostname and IP addresses are different.
> (so for replication, A has: tcps:hostname_of_b:12346 and B has tcps:hostname_of_a:12346). Also note that HAproxy is prepared but
> not in use at all.
> 
> # 2.2.33.1 (e9afa7f18): /usr/local/etc/dovecot/dovecot.conf
> # Pigeonhole version 0.4.20 (7cd71ba)
> # OS: FreeBSD 11.1-RELEASE amd64
> auth_mechanisms = plain login
> disable_plaintext_auth = no
> doveadm_password =  # hidden, use -P to show it
> haproxy_trusted_networks = IPv4_Haproxy IPv6_Haproxy
> lda_mailbox_autocreate = yes
> lda_mailbox_autosubscribe = yes
> lmtp_save_to_detail_mailbox = yes
> mail_fsync = always
> mail_location = sdbox:~/sdbox
> mail_plugins = " quota notify replication"
> managesieve_notify_capability = mailto
> managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext imapsieve vnd.dovecot.imapsieve
> namespace {
>  inbox = yes
>  location =
>  mailbox Drafts {
>    auto = subscribe
>    special_use = \Drafts
>  }
>  mailbox Junk {
>    special_use = \Junk
>  }
>  mailbox Sent {
>    auto = subscribe
>    special_use = \Sent
>  }
>  mailbox "Sent Messages" {
>    special_use = \Sent
>  }
>  mailbox Spam {
>    auto = subscribe
>    special_use = \Junk
>  }
>  mailbox Trash {
>    auto = subscribe
>    special_use = \Trash
>  }
>  prefix =
>  separator = .
> }
> passdb {
>  driver = pam
> }
> plugin {
>  imapsieve_mailbox1_before = file:/usr/local/lib/dovecot/sieve/report-spam.sieve
>  imapsieve_mailbox1_causes = COPY
>  imapsieve_mailbox1_name = Spam
>  imapsieve_mailbox2_before = file:/usr/local/lib/dovecot/sieve/report-ham.sieve
>  imapsieve_mailbox2_causes = COPY
>  imapsieve_mailbox2_from = Spam
>  imapsieve_mailbox2_name = *
>  mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename
>  mail_log_fields = uid box msgid size
>  mail_replica = tcps:hostname_of_machine_a:12346
>  sieve = ~/.dovecot.sieve
>  sieve_after = /usr/local/etc/dovecot/sieve/after/
>  sieve_before = /usr/local/etc/dovecot/sieve/global/
>  sieve_pipe_bin_dir = /usr/local/lib/dovecot/sieve
>  sieve_plugins = sieve_imapsieve sieve_extprograms
>  sieve_vacation_dont_check_recipient = yes
> }
> postmaster_address = postmaster at ourdomain.tld
> protocols = imap pop3 lmtp sieve
> replication_max_conns = 100
> service aggregator {
>  fifo_listener replication-notify-fifo {
>    mode = 0666
>  }
>  unix_listener replication-notify {
>    mode = 0666
>  }
> }
> service auth {
>  unix_listener /var/spool/postfix/private/auth {
>    mode = 0666
>  }
> }
> service doveadm {
>  inet_listener {
>    port = 12346
>    ssl = yes
>  }
> }
> service imap-login {
>  inet_listener imap_haproxy {
>    haproxy = yes
>    port = 10143
>  }
>  inet_listener imaps_haproxy {
>    haproxy = yes
>    port = 10144
>    ssl = yes
>  }
>  service_count = 1
> }
> service imap {
>  process_limit = 1024
> }
> service lmtp {
>  inet_listener lmtp {
>    address = IPv4, IPv6, 127.0.0.1 ::1
>    port = 24
>  }
>  unix_listener /var/spool/postfix/private/dovecot-lmtp {
>    group = postfix
>    mode = 0600
>    user = postfix
>  }
> }
> service pop3 {
>  process_limit = 1024
> }
> service replicator {
>  process_min_avail = 1
>  unix_listener replicator-doveadm {
>    mode = 0666
>  }
> }
> ssl_ca = </usr/local/etc/letsencrypt/live/host.ourdomain.tld/fullchain.pem
> ssl_cert = </usr/local/etc/letsencrypt/live/host.ourdomain.tld/cert.pem
> ssl_client_ca_file = /usr/local/certificates/letsencrypt-ca.pem
> ssl_key =  # hidden, use -P to show it
> ssl_protocols = !SSLv2 !SSLv3
> userdb {
>  driver = passwd
> }
> verbose_proctitle = yes
> protocol lmtp {
>  auth_username_format = %n
>  mail_plugins = quota sieve replication notify
>  postmaster_address = postmaster at ourdomain.tld
> }
> protocol lda {
>  mail_plugins = " quota notify replication sieve replication notify"
> }
> protocol imap {
>  imap_client_workarounds = delay-newmail
>  mail_max_userip_connections = 50
>  mail_plugins = " quota notify replication imap_quota imap_sieve"
> }
> 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: Message signed with OpenPGP
URL: <https://dovecot.org/pipermail/dovecot/attachments/20171020/c96d5f1e/attachment.sig>