[Dovecot] Problem with dsync replication : mails are not deleted correctly
Hello,
I would like to configure a postfix / dovecot cluster based on ssh / dsync replication.
I have two linux serveurs (Oracle RHEL 5 86_64) and I install both sides dovecot (V2.2.2+), postfix (2.10.0) and mysql (Ver 14.12 Distrib 5.0.77). I configured these servers in order to make them replicate, and I created some virtual users. I use some java code in order to send mail (with SMTP) and to get mail (with POP3). When the java code get a mail, it deletes it from mailbox (the flag "DELETED" is set to true).
When I send a mail on server A, I can see the mail replicated on both servers A and B : a file containing the message data exists in /home/vmail/test03/new/ in both sides. When I get the mail from server A with my java POP3 client, all is OK : the mail is deleted on both servers A and B (no more file in /home/vmail/test03/new/).
The problem is when I send a mail on server A and I get it from server B : I can get the mail, but the mail is not deleted from server B, and neither from server A. When I try a second time to get the mail on server B, then all becomes OK : the mail is deleted on both server A and B.
I can reproduce the problem every time.
Is it a known bug or not ? If yes, is there a patch ? If no, may the problem come from my config ?
Regards, Nicolas.
[root@int-proxy1 dovecot-2.2.2-p1]# doveconf -n # 2.2.2: /usr/local/etc/dovecot/dovecot.conf # OS: Linux 2.6.18-194.el5xen x86_64 Red Hat Enterprise Linux Server release 5.5 (Tikanga) ext3 disable_plaintext_auth = no hostname = int-proxy1 mail_location = maildir:/home/vmail/%u mail_plugins = acl quota notify replication mail_privileged_group = mail mbox_write_locks = fcntl namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { args = /usr/local/etc/dovecot/dovecot-sql.conf driver = sql } plugin { mail_replica = remote:vmail@int-proxy2 replication_full_sync_interval = 1 hours } pop3_uidl_format = %g postmaster_address = postmaster@mail.fluid-e.int service aggregator { fifo_listener replication-notify-fifo { mode = 0666 user = vmail } unix_listener replication-notify { mode = 0666 user = vmail } } service auth-worker { user = root } service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0666 user = postfix } unix_listener auth-userdb { mode = 0666 user = vmail } user = root } service config { unix_listener config { user = vmail } } service doveadm { user = vmail } service lmtp { unix_listener /var/spool/postfix/private/dovecot-lmtp { group = postfix mode = 0666 user = postfix } } service pop3-login { inet_listener pop3 { port = 110 } inet_listener pop3s { port = 995 ssl = yes } } service replicator { process_min_avail = 1 } ssl_cert = </etc/ssl/certs/dovecot.pem ssl_key = </etc/ssl/private/dovecot.pem submission_host = int-proxy1:25 userdb { args = /usr/local/etc/dovecot/dovecot-sql.conf driver = sql } protocol imap { mail_plugins = acl quota imap_acl imap_quota mail_log notify }
On 27.5.2013, at 17.09, Nicolas ROCHE <nicolas.roche@fluid-e.com> wrote:
I have two linux serveurs (Oracle RHEL 5 86_64) and I install both sides dovecot (V2.2.2+), postfix (2.10.0) and mysql (Ver 14.12 Distrib 5.0.77). I configured these servers in order to make them replicate, and I created some virtual users. I use some java code in order to send mail (with SMTP) and to get mail (with POP3). When the java code get a mail, it deletes it from mailbox (the flag "DELETED" is set to true).
When I send a mail on server A, I can see the mail replicated on both servers A and B : a file containing the message data exists in /home/vmail/test03/new/ in both sides.
The mail is delivered via LMTP?
When I get the mail from server A with my java POP3 client, all is OK : the mail is deleted on both servers A and B (no more file in /home/vmail/test03/new/).
The problem is when I send a mail on server A and I get it from server B : I can get the mail, but the mail is not deleted from server B, and neither from server A. When I try a second time to get the mail on server B, then all becomes OK : the mail is deleted on both server A and B.
I can reproduce the problem every time.
Is it a known bug or not ? If yes, is there a patch ? If no, may the problem come from my config ?
Shouldn't happen. Try this:
- Disable the replication plugin (remove from mail_plugins)
- Deliver a new mail on A
- Sync the new mail to the replica: doveadm sync -u user@domain -d
- Read + delete mail on B (and verify it gets deleted from B)
- Sync the deletion to the replica with rawlog enabled: doveadm sync -r rawlog -u user@domain -d
Now assuming that the mail was added back to B instead of being deleted from A, send the rawlog to me.
I have two linux serveurs (Oracle RHEL 5 86_64) and I install both sides dovecot (V2.2.2+), postfix (2.10.0) and mysql (Ver 14.12 Distrib 5.0.77). I configured these servers in order to make them replicate, and I created some virtual users. I use some java code in order to send mail (with SMTP) and to get mail (with POP3). When the java code get a mail, it deletes it from mailbox (the flag "DELETED" is set to true).
When I send a mail on server A, I can see the mail replicated on both servers A and B : a file containing the message data exists in /home/vmail/test03/new/ in both sides.
The mail is delivered via LMTP? ==> I don't think so (but I don't know exactly what is LMTP).
When I get the mail from server A with my java POP3 client, all is OK : the mail is deleted on both servers A and B (no more file in /home/vmail/test03/new/).
The problem is when I send a mail on server A and I get it from server B : I can get the mail, but the mail is not deleted from server B, and neither from server A. When I try a second time to get the mail on server B, then all becomes OK : the mail is deleted on both server A and B.
I can reproduce the problem every time.
Is it a known bug or not ? If yes, is there a patch ? If no, may the problem come from my config ?
Shouldn't happen. Try this:
Disable the replication plugin (remove from mail_plugins)
Deliver a new mail on A
Sync the new mail to the replica: doveadm sync -u user@domain -d
Read + delete mail on B (and verify it gets deleted from B)
Sync the deletion to the replica with rawlog enabled: doveadm sync -r rawlog -u user@domain -d
Now assuming that the mail was added back to B instead of being deleted from A, send the rawlog to me.
I tried the 5 steps successfully : 1 - I commented "mail_plugins" in my config and restart dovecot service 2 - I delivered I mail on server A : I can see the mail on the server A in /home/vmail/test03/new/ but not in server B. 3 - I synchronized servers : from server A I executed the command "doveadm sync -u test03 test03@int-proxy2". The mail is then in /home/vmail/test03/new/ on both servers A and B. 4 - I read and delete mail on server B. The mail is deleted from server B but is still on server A in /home/vmail/test03/new/. 5 - I synchronized servers : from server B I executed the command "doveadm sync -r rawlog -u test03 test03@int-proxy1". The mail is then deleted on both servers.
On 27.5.2013, at 17.50, Nicolas ROCHE <nicolas.roche@fluid-e.com> wrote:
I have two linux serveurs (Oracle RHEL 5 86_64) and I install both sides dovecot (V2.2.2+), postfix (2.10.0) and mysql (Ver 14.12 Distrib 5.0.77). I configured these servers in order to make them replicate, and I created some virtual users. I use some java code in order to send mail (with SMTP) and to get mail (with POP3). When the java code get a mail, it deletes it from mailbox (the flag "DELETED" is set to true).
When I send a mail on server A, I can see the mail replicated on both servers A and B : a file containing the message data exists in /home/vmail/test03/new/ in both sides.
The mail is delivered via LMTP? ==> I don't think so (but I don't know exactly what is LMTP).
How is Postfix configured to deliver the mail to Maildir? If it's not via Dovecot LDA or LMTP, that might be the cause of your problems.
On 27.5.2013, at 17.50, Nicolas ROCHE <nicolas.roche@fluid-e.com> wrote:
I have two linux serveurs (Oracle RHEL 5 86_64) and I install both sides dovecot (V2.2.2+), postfix (2.10.0) and mysql (Ver 14.12 Distrib 5.0.77). I configured these servers in order to make them replicate, and I created some virtual users. I use some java code in order to send mail (with SMTP) and to get mail (with POP3). When the java code get a mail, it deletes it from mailbox (the flag "DELETED" is set to true).
When I send a mail on server A, I can see the mail replicated on both servers A and B : a file containing the message data exists in /home/vmail/test03/new/ in both sides.
The mail is delivered via LMTP? ==> I don't think so (but I don't know exactly what is LMTP).
How is Postfix configured to deliver the mail to Maildir? If it's not via Dovecot LDA or LMTP, that might be the cause of your problems.
I checked my config and I can say that I am using dovecot LDA :
In /etc/postfix/main.cf I use "virtual_transport = dovecot" In /etc/postfix/master.cf I use : dovecot unix - n n - - pipe flags=DRhu user=vmail:vmail argv=/usr/local/libexec/dovecot/dovecot-lda -f ${sender} -d ${user}
I confirm I do not use LMTP : I removed all LMTP instructions from my dovecot config and I still have the same symptoms when I delete a message...
Is postfix used when a mail is deleted ? Is not it only dovecot (and dsync utility) who is used to synchronise the mail deletion ?
On 2013-05-27 11:33 AM, Nicolas ROCHE <nicolas.roche@fluid-e.com> wrote:
I checked my config and I can say that I am using dovecot LDA :
In /etc/postfix/main.cf I use "virtual_transport = dovecot" In /etc/postfix/master.cf I use : dovecot unix - n n - - pipe flags=DRhu user=vmail:vmail argv=/usr/local/libexec/dovecot/dovecot-lda -f ${sender} -d ${user}
I confirm I do not use LMTP : I removed all LMTP instructions from my dovecot config and I still have the same symptoms when I delete a message...
Is postfix used when a mail is deleted ? Is not it only dovecot (and dsync utility) who is used to synchronise the mail deletion ?
Please provide logs of a successful delivery showing that dovecot KDA is used.
Also, please don't show copy/snips of config files, always only show doveconf -n (or postconf -n for postfix) output - this makes sure that you are using the config that you think you are using.
--
Best regards,
Charles
These are the logs and config I can give you at the moment. I suppose it is possible to configure more verbose logs : I will search how to...
Regards, Nicolas.
====================================== When I send a mail on serveur A :
Server A : /var/log/maillog :
May 27 18:20:41 tmpl-vm03 postfix/smtpd[14476]: warning: dict_nis_init: NIS domain name not set - NIS lookups disabled May 27 18:20:41 tmpl-vm03 postfix/smtpd[14476]: warning: smtpd_sasl_auth_enable is true, but SASL support is not compiled in May 27 18:20:41 tmpl-vm03 postfix/smtpd[14476]: connect from unknown[192.168.0.70] May 27 18:20:41 tmpl-vm03 postfix/smtpd[14476]: 6FB077E415B: client=unknown[192.168.0.70] May 27 18:20:41 tmpl-vm03 postfix/cleanup[14482]: 6FB077E415B: message-id=<7866553.0.1369671641306.JavaMail.Administrateur@NICOLAS-ROCHE-P> May 27 18:20:41 tmpl-vm03 postfix/qmgr[30415]: 6FB077E415B: from=<nicolas.roche@fluid-e.com>, size=504, nrcpt=1 (queue active) May 27 18:20:41 tmpl-vm03 postfix/smtpd[14476]: disconnect from unknown[192.168.0.70] May 27 18:20:41 tmpl-vm03 dovecot: lda(test03): msgid=<7866553.0.1369671641306.JavaMail.Administrateur@NICOLAS-ROCHE-P>: saved mail to INBOX May 27 18:20:41 tmpl-vm03 postfix/pipe[14484]: 6FB077E415B: to=<test03@mail.fluid-e.int>, relay=dovecot, delay=0.16, delays=0.08/0.01/0/0.08, dsn=2.0.0, status=sent (delivered via dovecot service) May 27 18:20:41 tmpl-vm03 postfix/qmgr[30415]: 6FB077E415B: removed
Server B : /var/log/maillog :
No log.
====================================== When I get a mail on serveur B (first time) :
Server A : /var/log/maillog :
No log.
Server B : /var/log/maillog :
May 27 18:23:14 tmpl-vm03 dovecot: pop3-login: Login: user=<test03>, method=PLAIN, rip=192.168.0.70, lip=10.254.2.61, mpid=15472, session=</APAkLXdEADAqABG> May 27 18:23:14 tmpl-vm03 dovecot: pop3(test03): Disconnected: Logged out top=1/580, retr=0/0, del=1/1, size=585
====================================== When I get a mail on serveur B (second time) :
Server A : /var/log/maillog :
No log.
Server B : /var/log/maillog :
May 27 18:24:15 tmpl-vm03 dovecot: pop3-login: Login: user=<test03>, method=PLAIN, rip=192.168.0.70, lip=10.254.2.61, mpid=15479, session=<n/lflLXdnQDAqABG> May 27 18:24:15 tmpl-vm03 dovecot: pop3(test03): Disconnected: Logged out top=1/580, retr=0/0, del=1/1, size=585
================================= doveconf -n
[root@int-proxy1 ~]# doveconf -n # 2.2.2: /usr/local/etc/dovecot/dovecot.conf # OS: Linux 2.6.18-194.el5xen x86_64 Red Hat Enterprise Linux Server release 5.5 (Tikanga) ext3 disable_plaintext_auth = no hostname = int-proxy1 mail_location = maildir:/home/vmail/%u mail_plugins = acl quota notify replication mail_privileged_group = mail mbox_write_locks = fcntl namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { args = /usr/local/etc/dovecot/dovecot-sql.conf driver = sql } plugin { mail_replica = remote:vmail@int-proxy2 replication_full_sync_interval = 1 hours } pop3_uidl_format = %g postmaster_address = postmaster@mail.fluid-e.int protocols = imap pop3 service aggregator { fifo_listener replication-notify-fifo { mode = 0666 user = vmail } unix_listener replication-notify { mode = 0666 user = vmail } } service auth-worker { user = root } service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0666 user = postfix } unix_listener auth-userdb { mode = 0666 user = vmail } user = root } service config { unix_listener config { user = vmail } } service doveadm { user = vmail } service pop3-login { inet_listener pop3 { port = 110 } inet_listener pop3s { port = 995 ssl = yes } } service replicator { process_min_avail = 1 } ssl_cert = </etc/ssl/certs/dovecot.pem ssl_key = </etc/ssl/private/dovecot.pem submission_host = int-proxy1:25 userdb { args = /usr/local/etc/dovecot/dovecot-sql.conf driver = sql } protocol imap { mail_plugins = acl quota imap_acl imap_quota mail_log notify }
===================================== postconf -n
config_directory = /etc/postfix dovecot_destination_recipient_limit = 1 inet_interfaces = all mydestination = $myhostname localhost.$mydomain localhost $mydomain mynetworks = 127.0.0.0/8 10.253.0.0/16 10.254.0.0/16 10.255.0.0/16 192.168.0.0/16 myorigin = $mydomain relayhost = smtpd_recipient_restrictions = permit_mynetworks, permit_sasl_authenticated, reject_unauth_destination smtpd_sasl_auth_enable = yes smtpd_sasl_path = private/auth smtpd_sasl_type = dovecot virtual_alias_maps = mysql:/etc/postfix/mysql_virtual_alias_maps.cf virtual_gid_maps = static:5000 virtual_mailbox_base = /home/vmail virtual_mailbox_domains = mail.fluid-e.int virtual_mailbox_limit = 51200000 virtual_mailbox_maps = mysql:/etc/postfix/mysql_virtual_mailbox_maps.cf virtual_minimum_uid = 5000 virtual_transport = dovecot virtual_uid_maps = static:5000 postconf: warning: /etc/postfix/main.cf: unused parameter: virtual_overquota_bounce=yes postconf: warning: /etc/postfix/main.cf: unused parameter: virtual_maildir_limit_message=Sorry, the your maildir has overdrawn your diskspace quota, please free up some of spaces of your mailbox try again. postconf: warning: /etc/postfix/main.cf: unused parameter: virtual_mailbox_limit_maps=mysql:/etc/postfix/mysql_virtual_mailbox_limit_maps.cf postconf: warning: /etc/postfix/main.cf: unused parameter: virtual_mailbox_extended=yes postconf: warning: /etc/postfix/main.cf: unused parameter: virtual_mailbox_limit_override=yes postconf: warning: /etc/postfix/main.cf: unused parameter: virtual_create_maildirsize=yes
-----Message d'origine----- De : dovecot-bounces@dovecot.org [mailto:dovecot-bounces@dovecot.org] De la part de Charles Marcus Envoyé : lundi 27 mai 2013 18:17 À : dovecot@dovecot.org Objet : Re: [Dovecot] Problem with dsync replication : mails are not deleted correctly
On 2013-05-27 11:33 AM, Nicolas ROCHE <nicolas.roche@fluid-e.com> wrote:
I checked my config and I can say that I am using dovecot LDA :
In /etc/postfix/main.cf I use "virtual_transport = dovecot" In /etc/postfix/master.cf I use : dovecot unix - n n - - pipe flags=DRhu user=vmail:vmail argv=/usr/local/libexec/dovecot/dovecot-lda -f ${sender} -d ${user}
I confirm I do not use LMTP : I removed all LMTP instructions from my dovecot config and I still have the same symptoms when I delete a message...
Is postfix used when a mail is deleted ? Is not it only dovecot (and dsync utility) who is used to synchronise the mail deletion ?
Please provide logs of a successful delivery showing that dovecot KDA is used.
Also, please don't show copy/snips of config files, always only show doveconf -n (or postconf -n for postfix) output - this makes sure that you are using the config that you think you are using.
--
Best regards,
Charles
participants (3)
-
Charles Marcus
-
Nicolas ROCHE
-
Timo Sirainen