Dsync induces redundant mail after rapid append/expunge repeatedly
I meet the problem about dsync with expunge. The problems is expunged mail would reappear after dsyncing and it is easily to reproduce.
Environment Settings:
- Two servers(serverA, serverB) both enabled plugin "*replication"*
- And more detail get by "dovecot -n" show below
Reproduce flow: - Run two processes pA and pB both would run in while loop - pA(All behavior is on Draft mailbox and all on serverA):
- Clean "Draft" mailbox of userA
- Run the following imap command of userA
- imap save (save new mail to Draft)
- imap delete (delete old mail on Draft)
- imap expunge
- Check if draft only have one email (it would be failed in about 50 loops)* - pB(Send mail to serverB):
- Send new mail to userA (which would be save in INBOX)
It would be easier reproduce by increasing server loading.
I have found something - T he expunge changes from server A in dsync connection don't send immediately, these expunge changes appear after 1~2 dsync connection over.
- The replicated mail is generated from serverB and sync back to serverA so expunged mail reappeared. dovecot -n (ServerA)
2.2.19: /etc/dovecot/dovecot.conf
# OS: Linux 3.10.77 x86_64 ext4 auth_mechanisms = plain login default_internal_user = root default_login_user = dovecot default_vsz_limit = 0 disable_plaintext_auth = no doveadm_password = # hidden, use -P to show it dsync_remote_cmd = ssh -l%{login} %{host} doveadm dsync-server -u%u dsync_target_servers = tcps:10.13.23.139:24245 lda_mailbox_autocreate = yes mail_debug = yes mail_fsync = never mail_location = maildir:/var/spool/mail/%d/%-4.04i/%i/Maildir mail_max_userip_connections = 30 mail_never_cache_fields = * mail_plugins = " notify replication mail_log" managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave duplicate namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { driver = pam } plugin { fts = lucene fts_autoindex = yes fts_lucene = whitespace_chars=@._/\~&,()[]" mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename flag_change append mail_log_fields = uid box msgid size flags from subject } postmaster_address = postmaster@localhost protocols = imap pop3 lmtp imap pop3 sieve replication_dsync_parameters = -N -l 30 -U service aggregator { fifo_listener replication-notify-fifo { group = users mode = 0666 user = admin } unix_listener replication-notify { group = users mode = 0666 user = admin } } service doveadm { inet_listener { address = 10.13.23.136,127.0.0.1 port = 24245 ssl = yes } } service imap-login { inet_listener imap { address = 10.13.23.136,127.0.0.1 port = 143 } inet_listener imaps { address = 10.13.23.136,127.0.0.1 port = 993 ssl = yes } process_limit = 256 service_count = 0 } service indexer-worker { process_limit = 8 } service pop3-login { inet_listener pop3 { address = 10.13.23.136,127.0.0.1 port = 110 } inet_listener pop3s { address = 10.13.23.136,127.0.0.1 port = 995 ssl = yes } process_limit = 256 service_count = 0 } service replicator { unix_listener replicator-doveadm { mode = 0666 } } userdb { driver = passwd } protocol lda { mail_plugins = " notify replication mail_log sieve" }
dovecot -n (ServerB) $ ./sbin/dovecot -n
# 2.2.19: /etc/dovecot/dovecot.conf # OS: Linux 3.10.77 x86_64 auth_mechanisms = plain login default_internal_user = root default_login_user = dovecot default_vsz_limit = 0 disable_plaintext_auth = no doveadm_password = # hidden, use -P to show it dsync_remote_cmd = ssh -l%{login} %{host} doveadm dsync-server -u%u dsync_target_servers = tcps:10.13.23.136:24245 lda_mailbox_autocreate = yes mail_debug = yes mail_fsync = always mail_location = maildir:/var/spool/mail/%d/%-4.04i/%i/Maildir mail_max_userip_connections = 30 mail_never_cache_fields = * mail_plugins = " notify replication mail_log" managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational reg ex imap4flags copy include variables body enotify environment mailbox date ihave duplicate namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { driver = pam } plugin { fts = lucene fts_autoindex = yes fts_lucene = whitespace_chars=@._/\~&,()[]" mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename flag_change append mail_log_fields = uid box msgid size flags from subject } postmaster_address = postmaster@localhost protocols = imap pop3 lmtp imap pop3 sieve replication_dsync_parameters = -N -l 30 -U service aggregator { fifo_listener replication-notify-fifo { group = users mode = 0666 user = admin } unix_listener replication-notify { group = users mode = 0666 user = admin } } service doveadm { inet_listener { address = 10.13.23.139,127.0.0.1 port = 24245 ssl = yes } } service imap-login { inet_listener imap { address = 10.13.23.139,127.0.0.1 port = 143 } inet_listener imaps { address = 10.13.23.139,127.0.0.1 port = 993 ssl = yes } process_limit = 256 service_count = 0 } service indexer-worker { process_limit = 4 } service pop3-login { inet_listener pop3 { address = 10.13.23.139,127.0.0.1 port = 110 } inet_listener pop3s { address = 10.13.23.139,127.0.0.1 port = 995 ssl = yes } process_limit = 256 service_count = 0 } service replicator { unix_listener replicator-doveadm { mode = 0666 } } userdb { driver = passwd } protocol lda { mail_plugins = " notify replication mail_log sieve" }
Regards, Jay Pan
On 01 Mar 2016, at 13:20, 潘信傑 sfffaaa@gmail.com wrote:
I meet the problem about dsync with expunge. The problems is expunged mail would reappear after dsyncing and it is easily to reproduce.
I've never been able to reproduce this easily. But I wonder if this fix would help here:
https://github.com/dovecot/core/commit/950a6e61d6c2dac961ce031bdd8b2895bc32b... https://github.com/dovecot/core/commit/950a6e61d6c2dac961ce031bdd8b2895bc32b...
Hi Timo,
Thanks for your reply, I've patched this commit before, but it cannot solve my problem.
Besides, I log more information on server B and it shows server B generates the mail itself. (The mail which has UID 2128 is the redundant mail and sync back to server A)
Feb 24 22:57:36 dsync-server(test 28740): Info: highest_seen_uid 2125, importer->local_uid_next 2124 Feb 24 22:57:36 dsync-server(test 28740): Debug: brain S: Import Drafts: unwanted UIDs: 2124:2125 Feb 24 22:57:36 dsync-server(test 28740): Debug: brain S: Import Drafts: Reassign UIDs: 2124
Feb 24 22:57:37 dsync-local(test 28748): Debug: brain M: Import Drafts: UID 2125 isn't usable, assigning new UID 2128 Feb 24 22:57:37 dsync-local(test 28748): Info: copy from Drafts: box=Drafts, uid=2128, msgid=20151009090132.62413F6136D@test.yoyo.mail, size=34744, from="wf skr" wfskr@fstml.us, subject=The wf skr..., flags=(\Seen) Feb 24 22:57:37 dsync-local(test 28748): Info: expunge: box=Drafts, uid=2125, msgid=20151009090132.62413F6136D@test.yoyo.mail, size=34744, from="wf skr" wfskr@fstml.us, subject=The wf skr..., flags=(\Seen)
Best Regards, Jay Pan
2016-03-01 19:24 GMT+08:00 Timo Sirainen timo.sirainen@dovecot.fi:
On 01 Mar 2016, at 13:20, 潘信傑 sfffaaa@gmail.com wrote:
I meet the problem about dsync with expunge. The problems is expunged mail would reappear after dsyncing and it is easily to reproduce.
I've never been able to reproduce this easily. But I wonder if this fix would help here:
https://github.com/dovecot/core/commit/950a6e61d6c2dac961ce031bdd8b2895bc32b...
participants (2)
-
Timo Sirainen
-
潘信傑