Duplicate mails with pop3 + dsync replication
Hi,
When I enable (and it only happens when it is actually running) replication over TCP my users get duplicate mails via pop3. Here’s what happens:
- User gets mail
- Mail appears in (Maildir) new/ folder on both replicas
- user retrieves mail (retr), calls dele. Mail still exists in new/ folder on both replicas.
- user runs retr (and gets the duplicate) and dele a second time, the mail is actually removed. If I disable replication this behaviour goes away. I use dovecot from debian stable resources. Any help would _really_ be appreciated. I found one german thread that dealt with the same problem, but no solution for it (http://www.dovecot-buch.de/mailingliste/pop3-fehler-bei-delete-und-dsync-rep...)
wogri
# dovecot --version 2.2.13
# dovecot -n # 2.2.13: /etc/dovecot/dovecot.conf # OS: Linux 3.16.0-4-amd64 x86_64 Debian 8.0 ext4 auth_verbose = yes doveadm_password = XXX first_valid_gid = 113 first_valid_uid = 108 last_valid_gid = 113 last_valid_uid = 108 mail_gid = dovecot mail_location = maildir:/var/mail/%n mail_plugins = quota fts fts_lucene virtual notify replication mail_temp_dir = /var/lib/dovecot/tmp mail_uid = dovecot namespace { list = children location = virtual:/var/mail/%n/virtual prefix = virtual. separator = . } namespace inbox { inbox = yes list = yes location = mailbox "Deleted Messages" { auto = subscribe special_use = \Trash } mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { auto = no special_use = \Sent } mailbox Spam { auto = subscribe special_use = \Junk } [48/4646] mailbox Trash { special_use = \Trash } prefix = separator = . subscriptions = yes type = private } passdb { args = /etc/dovecot/dovecot-ldap.conf.ext driver = ldap } plugin { default_language = de fts = lucene fts_lucene = whitespace_chars=@. mail_replica = tcp:172.16.1.1:12345 quota = maildir:User quota quota_rule = *:storage=4G quota_rule2 = Trash:storage=+100M quota_rule3 = Spam:ignore quota_warning = storage=95%% quota-warning 95 %u quota_warning2 = storage=80%% quota-warning 80 %u sieve = /etc/sieve/%n.sieve sieve_default = /etc/sieve/default.sieve sieve_dir = ~/sieve sieve_extensions = +editheader } postmaster_address = postmaster@wogri.at protocols = " imap lmtp pop3" service aggregator { fifo_listener replication-notify-fifo { user = dovecot } unix_listener replication-notify { user = dovecot } } service doveadm { inet_listener { port = 12345 } } service imap { process_limit = 1024 } } service lmtp { inet_listener lmtp { address = 127.0.0.1 port = 2003 } user = dovecot } service managesieve-login { inet_listener sieve { port = 4190 } service_count = 1 } service pop3 { process_limit = 1024 } service quota-warning { executable = script /usr/local/sbin/quota-warning.sh unix_listener quota-warning { user = dovecot } user = dovecot } service replicator { process_min_avail = 1 unix_listener replicator-doveadm { mode = 0600 } } ssl = required ssl_cert = </etc/exim4/server.cert.pem ssl_key = </etc/exim4/server.key.pem userdb { args = /etc/dovecot/dovecot-ldap.conf.ext driver = ldap } protocol lmtp { mail_plugins = quota sieve virtual notify replication } protocol lda { mail_plugins = quota fts fts_lucene virtual notify replication sieve } protocol imap { mail_max_userip_connections = 50 mail_plugins = quota fts fts_lucene virtual notify replication imap_quota }
FWIW I just tried the sdbox mailbox format for a singe user (not the whole installation) by overriding the mail-userattr via ldap and the problem persists, so I don’t think it’s an issue with maildir per se, but rather a pop3 / replication issue or a misconfiguration on my side.
Wolfgang
On 20 Jun 2015, at 22:29, Wolfgang Hennerbichler <wogri@wogri.com> wrote:
Hi,
When I enable (and it only happens when it is actually running) replication over TCP my users get duplicate mails via pop3. Here’s what happens:
- User gets mail
- Mail appears in (Maildir) new/ folder on both replicas
- user retrieves mail (retr), calls dele. Mail still exists in new/ folder on both replicas.
- user runs retr (and gets the duplicate) and dele a second time, the mail is actually removed. If I disable replication this behaviour goes away. I use dovecot from debian stable resources. Any help would _really_ be appreciated. I found one german thread that dealt with the same problem, but no solution for it (http://www.dovecot-buch.de/mailingliste/pop3-fehler-bei-delete-und-dsync-rep...)
wogri
# dovecot --version 2.2.13
# dovecot -n # 2.2.13: /etc/dovecot/dovecot.conf # OS: Linux 3.16.0-4-amd64 x86_64 Debian 8.0 ext4 auth_verbose = yes doveadm_password = XXX first_valid_gid = 113 first_valid_uid = 108 last_valid_gid = 113 last_valid_uid = 108 mail_gid = dovecot mail_location = maildir:/var/mail/%n mail_plugins = quota fts fts_lucene virtual notify replication mail_temp_dir = /var/lib/dovecot/tmp mail_uid = dovecot namespace { list = children location = virtual:/var/mail/%n/virtual prefix = virtual. separator = . } namespace inbox { inbox = yes list = yes location = mailbox "Deleted Messages" { auto = subscribe special_use = \Trash } mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { auto = no special_use = \Sent } mailbox Spam { auto = subscribe special_use = \Junk } [48/4646] mailbox Trash { special_use = \Trash } prefix = separator = . subscriptions = yes type = private } passdb { args = /etc/dovecot/dovecot-ldap.conf.ext driver = ldap } plugin { default_language = de fts = lucene fts_lucene = whitespace_chars=@. mail_replica = tcp:172.16.1.1:12345 quota = maildir:User quota quota_rule = *:storage=4G quota_rule2 = Trash:storage=+100M quota_rule3 = Spam:ignore quota_warning = storage=95%% quota-warning 95 %u quota_warning2 = storage=80%% quota-warning 80 %u sieve = /etc/sieve/%n.sieve sieve_default = /etc/sieve/default.sieve sieve_dir = ~/sieve sieve_extensions = +editheader } postmaster_address = postmaster@wogri.at protocols = " imap lmtp pop3" service aggregator { fifo_listener replication-notify-fifo { user = dovecot } unix_listener replication-notify { user = dovecot } } service doveadm { inet_listener { port = 12345 } } service imap { process_limit = 1024 } } service lmtp { inet_listener lmtp { address = 127.0.0.1 port = 2003 } user = dovecot } service managesieve-login { inet_listener sieve { port = 4190 } service_count = 1 } service pop3 { process_limit = 1024 } service quota-warning { executable = script /usr/local/sbin/quota-warning.sh unix_listener quota-warning { user = dovecot } user = dovecot } service replicator { process_min_avail = 1 unix_listener replicator-doveadm { mode = 0600 } } ssl = required ssl_cert = </etc/exim4/server.cert.pem ssl_key = </etc/exim4/server.key.pem userdb { args = /etc/dovecot/dovecot-ldap.conf.ext driver = ldap } protocol lmtp { mail_plugins = quota sieve virtual notify replication } protocol lda { mail_plugins = quota fts fts_lucene virtual notify replication sieve } protocol imap { mail_max_userip_connections = 50 mail_plugins = quota fts fts_lucene virtual notify replication imap_quota }
It turns out that if I enable this option: pop3_deleted_flag = "$POP3Deleted”
The issue no longer persists. I have to manually expunge the kept mails that have been deleted via pop3 though: doveadm expunge mailbox INBOX KEYWORD '$POP3Deleted' -A
Wolfgang
On 21 Jun 2015, at 21:05, Wolfgang Hennerbichler <wogri@wogri.com> wrote:
FWIW I just tried the sdbox mailbox format for a singe user (not the whole installation) by overriding the mail-userattr via ldap and the problem persists, so I don’t think it’s an issue with maildir per se, but rather a pop3 / replication issue or a misconfiguration on my side.
Wolfgang
On 20 Jun 2015, at 22:29, Wolfgang Hennerbichler <wogri@wogri.com> wrote:
Hi,
When I enable (and it only happens when it is actually running) replication over TCP my users get duplicate mails via pop3. Here’s what happens:
- User gets mail
- Mail appears in (Maildir) new/ folder on both replicas
- user retrieves mail (retr), calls dele. Mail still exists in new/ folder on both replicas.
- user runs retr (and gets the duplicate) and dele a second time, the mail is actually removed. If I disable replication this behaviour goes away. I use dovecot from debian stable resources. Any help would _really_ be appreciated. I found one german thread that dealt with the same problem, but no solution for it (http://www.dovecot-buch.de/mailingliste/pop3-fehler-bei-delete-und-dsync-rep...)
wogri
# dovecot --version 2.2.13
# dovecot -n # 2.2.13: /etc/dovecot/dovecot.conf # OS: Linux 3.16.0-4-amd64 x86_64 Debian 8.0 ext4 auth_verbose = yes doveadm_password = XXX first_valid_gid = 113 first_valid_uid = 108 last_valid_gid = 113 last_valid_uid = 108 mail_gid = dovecot mail_location = maildir:/var/mail/%n mail_plugins = quota fts fts_lucene virtual notify replication mail_temp_dir = /var/lib/dovecot/tmp mail_uid = dovecot namespace { list = children location = virtual:/var/mail/%n/virtual prefix = virtual. separator = . } namespace inbox { inbox = yes list = yes location = mailbox "Deleted Messages" { auto = subscribe special_use = \Trash } mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { auto = no special_use = \Sent } mailbox Spam { auto = subscribe special_use = \Junk } [48/4646] mailbox Trash { special_use = \Trash } prefix = separator = . subscriptions = yes type = private } passdb { args = /etc/dovecot/dovecot-ldap.conf.ext driver = ldap } plugin { default_language = de fts = lucene fts_lucene = whitespace_chars=@. mail_replica = tcp:172.16.1.1:12345 quota = maildir:User quota quota_rule = *:storage=4G quota_rule2 = Trash:storage=+100M quota_rule3 = Spam:ignore quota_warning = storage=95%% quota-warning 95 %u quota_warning2 = storage=80%% quota-warning 80 %u sieve = /etc/sieve/%n.sieve sieve_default = /etc/sieve/default.sieve sieve_dir = ~/sieve sieve_extensions = +editheader } postmaster_address = postmaster@wogri.at protocols = " imap lmtp pop3" service aggregator { fifo_listener replication-notify-fifo { user = dovecot } unix_listener replication-notify { user = dovecot } } service doveadm { inet_listener { port = 12345 } } service imap { process_limit = 1024 } } service lmtp { inet_listener lmtp { address = 127.0.0.1 port = 2003 } user = dovecot } service managesieve-login { inet_listener sieve { port = 4190 } service_count = 1 } service pop3 { process_limit = 1024 } service quota-warning { executable = script /usr/local/sbin/quota-warning.sh unix_listener quota-warning { user = dovecot } user = dovecot } service replicator { process_min_avail = 1 unix_listener replicator-doveadm { mode = 0600 } } ssl = required ssl_cert = </etc/exim4/server.cert.pem ssl_key = </etc/exim4/server.key.pem userdb { args = /etc/dovecot/dovecot-ldap.conf.ext driver = ldap } protocol lmtp { mail_plugins = quota sieve virtual notify replication } protocol lda { mail_plugins = quota fts fts_lucene virtual notify replication sieve } protocol imap { mail_max_userip_connections = 50 mail_plugins = quota fts fts_lucene virtual notify replication imap_quota }
On 22/6/2015 9:49 am, Wolfgang Hennerbichler wrote:
It turns out that if I enable this option: pop3_deleted_flag = "$POP3Deleted”
The issue no longer persists. I have to manually expunge the kept mails that have been deleted via pop3 though: doveadm expunge mailbox INBOX KEYWORD '$POP3Deleted' -A
Hello,
I was wondering if the same behavior persists with latest version(s), for example 2.2.18.
If so, how do you overcome this issue? What is the suggested practice?
Can someone please provide administration guidelines to handle it?
I am seeing this behavior on v2.2.13; I am now planning an upgrade to 2.2.18.
Please advise!
My setup is:
Master 1:
dsync_remote_cmd = ssh -l root master2.example.com doveadm
dsync-server -u%u
replication_dsync_parameters = -d -N -l 30 -U
service aggregator {
fifo_listener replication-notify-fifo {
user = vmail
}
unix_listener replication-notify {
user = vmail
}
}
service replicator {
unix_listener replicator-doveadm {
mode = 0600
}
}Master 2:
dsync_remote_cmd = ssh -l root master1.example.com doveadm
dsync-server -u%u
replication_dsync_parameters = -d -N -l 30 -U
service aggregator {
fifo_listener replication-notify-fifo {
user = vmail
}
unix_listener replication-notify {
user = vmail
}
}
service replicator {
unix_listener replicator-doveadm {
mode = 0600
}
}Replication plugin is enabled on protocols pop3, imap, lda.
Thanks in advance, Nick
participants (2)
-
Nikolaos Milas
-
Wolfgang Hennerbichler