replication and spam removal ("doveadm expunge")
Hello everybody, since now I did no replication and spam is delivered into users folder "spambox" Every night there is a cronjob which deletes spam older than 30 days via something like "find .... -ctime +30 -delete" Now I'm going to set up replication (two way) and I thought that doing "rm" is not a good idea. So I modified the job to something like /usr/bin/doveadm expunge -u test1 mailbox INBOX.spambox BEFORE 30d which works like intended, but I see that on the replication partner the spam isn't deleted. Even if I do a doveadm force-resync -u test1 '*' or log in via imap to the replication partner I still see the old spam.
So my question is: is this the intended behaviour and I have to run the "doveadm expunge" on both replication partners or should the deletion via "expunge" being replicated to the partner and I have a misconfiguration or maybe I hit a bug ?
Regards, Olaf
/etc/dovecot/conf.d/12-replication.conf:
mail_plugins = $mail_plugins notify replication service aggregator { fifo_listener replication-notify-fifo { mode = 0666 } unix_listener replication-notify { mode = 0666 } } service replicator { process_min_avail = 1 unix_listener replicator-doveadm { mode = 0666 } } service doveadm { inet_listener { port = 1109 } } doveadm_password = XXXXX plugin { mail_replica = tcp:X.Y.Z.X:1109 }
-- Karlsruher Institut für Technologie (KIT) ATIS - Abt. Technische Infrastruktur, Fakultät für Informatik
Dipl.-Geophys. Olaf Hopp
- Leitung IT-Dienste -
Am Fasanengarten 5, Gebäude 50.34, Raum 009 76131 Karlsruhe Telefon: +49 721 608-43973 Fax: +49 721 608-46699 E-Mail: Olaf.Hopp@kit.edu www.atis.informatik.kit.edu
www.kit.edu
KIT - Die Forschungsuniversität in der Helmholtz-Gemeinschaft
Das KIT ist seit 2010 als familiengerechte Hochschule zertifiziert.
Nobody ? :-(
On 3/30/20 5:26 PM, Olaf Hopp wrote:
Hello everybody, since now I did no replication and spam is delivered into users folder "spambox" Every night there is a cronjob which deletes spam older than 30 days via something like "find .... -ctime +30 -delete" Now I'm going to set up replication (two way) and I thought that doing "rm" is not a good idea. So I modified the job to something like /usr/bin/doveadm expunge -u test1 mailbox INBOX.spambox BEFORE 30d which works like intended, but I see that on the replication partner the spam isn't deleted. Even if I do a doveadm force-resync -u test1 '*' or log in via imap to the replication partner I still see the old spam.
So my question is: is this the intended behaviour and I have to run the "doveadm expunge" on both replication partners or should the deletion via "expunge" being replicated to the partner and I have a misconfiguration or maybe I hit a bug ?
Regards, Olaf
/etc/dovecot/conf.d/12-replication.conf:
mail_plugins = $mail_plugins notify replication service aggregator { fifo_listener replication-notify-fifo { mode = 0666 } unix_listener replication-notify { mode = 0666 } } service replicator { process_min_avail = 1 unix_listener replicator-doveadm { mode = 0666 } } service doveadm { inet_listener { port = 1109 } } doveadm_password = XXXXX plugin { mail_replica = tcp:X.Y.Z.X:1109 }
-- Karlsruher Institut für Technologie (KIT) ATIS - Abt. Technische Infrastruktur, Fakultät für Informatik
Dipl.-Geophys. Olaf Hopp
- Leitung IT-Dienste -
Am Fasanengarten 5, Gebäude 50.34, Raum 009 76131 Karlsruhe Telefon: +49 721 608-43973 Fax: +49 721 608-46699 E-Mail: Olaf.Hopp@kit.edu www.atis.informatik.kit.edu
www.kit.edu
KIT - Die Forschungsuniversität in der Helmholtz-Gemeinschaft
Das KIT ist seit 2010 als familiengerechte Hochschule zertifiziert.
Hi Aki,
On 4/4/20 8:12 PM, Aki Tuomi wrote:
Can you provide doveconf -n and try turning on mail_debug=yes on both ends and try doveadm -Dv expunge ....
mail_debug=yes is on on both ends and dovecot was restarted but anyway nothing is logged when I issue "doveadm -Dv expunge " In the shell where I issue the "expunge" I see the following:
# /usr/bin/doveadm -Dv expunge -u test4 mailbox INBOX.spambox BEFORE 13m Debug: Loading modules from directory: /usr/lib64/dovecot Debug: Module loaded: /usr/lib64/dovecot/lib15_notify_plugin.so Debug: Module loaded: /usr/lib64/dovecot/lib20_replication_plugin.so Debug: Loading modules from directory: /usr/lib64/dovecot/doveadm Debug: Skipping module doveadm_acl_plugin, because dlopen() failed: /usr/lib64/dovecot/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol: acl_user_module (this is usually intentional, so just ignore this message) Debug: Skipping module doveadm_expire_plugin, because dlopen() failed: /usr/lib64/dovecot/doveadm/lib10_doveadm_expire_plugin.so: undefined symbol: expire_set_deinit (this is usually intentional, so just ignore this message) Debug: Skipping module doveadm_quota_plugin, because dlopen() failed: /usr/lib64/dovecot/doveadm/lib10_doveadm_quota_plugin.so: undefined symbol: quota_user_module (this is usually intentional, so just ignore this message) Debug: Module loaded: /usr/lib64/dovecot/doveadm/lib10_doveadm_sieve_plugin.so Debug: Skipping module doveadm_fts_lucene_plugin, because dlopen() failed: /usr/lib64/dovecot/doveadm/lib20_doveadm_fts_lucene_plugin.so: undefined symbol: lucene_index_iter_deinit (this is usually intentional, so just ignore this message) Debug: Skipping module doveadm_fts_plugin, because dlopen() failed: /usr/lib64/dovecot/doveadm/lib20_doveadm_fts_plugin.so: undefined symbol: fts_user_get_language_list (this is usually intentional, so just ignore this message) Debug: Skipping module doveadm_mail_crypt_plugin, because dlopen() failed: /usr/lib64/dovecot/doveadm/libdoveadm_mail_crypt_plugin.so: undefined symbol: mail_crypt_box_get_pvt_digests (this is usually intentional, so just ignore this message) doveadm(test4)<19830><>: Debug: auth-master: userdb lookup(test4): Started userdb lookup doveadm(test4)<19830><>: Debug: auth-master: conn unix:/var/run/dovecot/auth-userdb: Connecting doveadm(test4)<19830><>: Debug: auth-master: conn unix:/var/run/dovecot/auth-userdb (pid=15116,uid=0): Client connected (fd=10) doveadm(test4)<19830><>: Debug: auth-master: userdb lookup(test4): auth USER input: test4 system_groups_user=test4 uid=1805 gid=2300 home=/home/irams1-test/test4 doveadm(test4)<19830><>: Debug: auth-master: userdb lookup(test4): Finished userdb lookup (username=test4 system_groups_user=test4 uid=1805 gid=2300 home=/home/irams1-test/test4) doveadm(test4): Debug: Effective uid=1805, gid=2300, home=/home/irams1-test/test4 doveadm(test4): Debug: Namespace inbox: type=private, prefix=INBOX., sep=., inbox=yes, hidden=no, list=yes, subscriptions=yes location=maildir:~/Maildir doveadm(test4): Debug: maildir++: root=/home/irams1-test/test4/Maildir, index=, indexpvt=, control=, inbox=/home/irams1-test/test4/Maildir, alt= doveadm(test4): Debug: Namespace : type=private, prefix=, sep=, inbox=no, hidden=yes, list=no, subscriptions=no location=fail::LAYOUT=none doveadm(test4): Debug: none: root=, index=, indexpvt=, control=, inbox=, alt= doveadm(test4): Debug: Mailbox INBOX.spambox: Mailbox opened because: expunge doveadm(test4): Debug: expunge: box=INBOX.spambox uid=38 doveadm(test4): Debug: expunge: box=INBOX.spambox uid=39 doveadm(test4): Debug: expunge: box=INBOX.spambox uid=40 doveadm(test4): Debug: expunge: box=INBOX.spambox uid=41 doveadm(test4): Debug: expunge: box=INBOX.spambox uid=42 doveadm(test4): Debug: expunge: box=INBOX.spambox uid=43 doveadm(test4): Debug: expunge: box=INBOX.spambox uid=44 doveadm(test4): Debug: expunge: box=INBOX.spambox uid=45 doveadm(test4): Debug: expunge: box=INBOX.spambox uid=46 doveadm(test4): Debug: auth-master: conn unix:/var/run/dovecot/auth-userdb (pid=15116,uid=0): Disconnected: Connection closed (fd=10)
A few mails are deleted from the spambox but the deletion is not synced to the other side Even if I do a "doveadm force-resync -u test4 '*'" on both sides the deletions are not replicated but anyway
#doveadm replicator status test4 username priority fast sync full sync success sync failed test4 none 00:26:47 18:19:46 00:26:47 -
on both sides. If new spam arrives it got well replicated to the other side as expected
"doveconf -n" ist attached below.
Thanks, Olaf
Aki
On 04/04/2020 20:03 Olaf Hopp < olaf.hopp@kit.edu mailto:olaf.hopp@kit.edu> wrote:
Nobody ? :-(
On 3/30/20 5:26 PM, Olaf Hopp wrote:
Hello everybody, since now I did no replication and spam is delivered into users folder "spambox" Every night there is a cronjob which deletes spam older than 30 days via something like "find .... -ctime +30 -delete" Now I'm going to set up replication (two way) and I thought that doing "rm" is not a good idea. So I modified the job to something like /usr/bin/doveadm expunge -u test1 mailbox INBOX.spambox BEFORE 30d which works like intended, but I see that on the replication partner the spam isn't deleted. Even if I do a doveadm force-resync -u test1 '*' or log in via imap to the replication partner I still see the old spam.
So my question is: is this the intended behaviour and I have to run the "doveadm expunge" on both replication partners or should the deletion via "expunge" being replicated to the partner and I have a misconfiguration or maybe I hit a bug ?
Regards, Olaf
/etc/dovecot/conf.d/12-replication.conf:
mail_plugins = $mail_plugins notify replication service aggregator { fifo_listener replication-notify-fifo { mode = 0666 } unix_listener replication-notify { mode = 0666 } } service replicator { process_min_avail = 1 unix_listener replicator-doveadm { mode = 0666 } } service doveadm { inet_listener { port = 1109 } } doveadm_password = XXXXX plugin { mail_replica = tcp:X.Y.Z.X:1109 }
-- Karlsruher Institut für Technologie (KIT) ATIS - Abt. Technische Infrastruktur, Fakultät für Informatik
Dipl.-Geophys. Olaf Hopp
- Leitung IT-Dienste -
Am Fasanengarten 5, Gebäude 50.34, Raum 009 76131 Karlsruhe Telefon: +49 721 608-43973 Fax: +49 721 608-46699 E-Mail: Olaf.Hopp@kit.edu mailto:Olaf.Hopp@kit.edu www.atis.informatik.kit.edu
www.kit.edu
KIT - Die Forschungsuniversität in der Helmholtz-Gemeinschaft
Das KIT ist seit 2010 als familiengerechte Hochschule zertifiziert.
Aki Tuomi
-- Karlsruher Institut für Technologie (KIT) ATIS - Abt. Technische Infrastruktur, Fakultät für Informatik
Dipl.-Geophys. Olaf Hopp
- Leitung IT-Dienste -
Am Fasanengarten 5, Gebäude 50.34, Raum 009 76131 Karlsruhe Telefon: +49 721 608-43973 Fax: +49 721 608-46699 E-Mail: Olaf.Hopp@kit.edu www.atis.informatik.kit.edu
www.kit.edu
KIT - Die Forschungsuniversität in der Helmholtz-Gemeinschaft
Das KIT ist seit 2010 als familiengerechte Hochschule zertifiziert.
OK, further findings: when user logs in on the node where the "doveadm expunge" was run this has no effect on the spambox the other side.
But if the user logs in on the other side (after a manual failover of the "cluster" IP) and opens his spambox then all of a sudden those expunged mails got deleted.
So if this behaviour is the intended then you may regard this issue as closed.
Thanks, Olaf
On 4/6/20 12:40 PM, Olaf Hopp wrote:
Hi Aki,
On 4/4/20 8:12 PM, Aki Tuomi wrote:
Can you provide doveconf -n and try turning on mail_debug=yes on both ends and try doveadm -Dv expunge ....
mail_debug=yes is on on both ends and dovecot was restarted but anyway nothing is logged when I issue "doveadm -Dv expunge " In the shell where I issue the "expunge" I see the following:
# /usr/bin/doveadm -Dv expunge -u test4 mailbox INBOX.spambox BEFORE 13m Debug: Loading modules from directory: /usr/lib64/dovecot Debug: Module loaded: /usr/lib64/dovecot/lib15_notify_plugin.so Debug: Module loaded: /usr/lib64/dovecot/lib20_replication_plugin.so Debug: Loading modules from directory: /usr/lib64/dovecot/doveadm Debug: Skipping module doveadm_acl_plugin, because dlopen() failed: /usr/lib64/dovecot/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol: acl_user_module (this is usually intentional, so just ignore this message) Debug: Skipping module doveadm_expire_plugin, because dlopen() failed: /usr/lib64/dovecot/doveadm/lib10_doveadm_expire_plugin.so: undefined symbol: expire_set_deinit (this is usually intentional, so just ignore this message) Debug: Skipping module doveadm_quota_plugin, because dlopen() failed: /usr/lib64/dovecot/doveadm/lib10_doveadm_quota_plugin.so: undefined symbol: quota_user_module (this is usually intentional, so just ignore this message) Debug: Module loaded: /usr/lib64/dovecot/doveadm/lib10_doveadm_sieve_plugin.so Debug: Skipping module doveadm_fts_lucene_plugin, because dlopen() failed: /usr/lib64/dovecot/doveadm/lib20_doveadm_fts_lucene_plugin.so: undefined symbol: lucene_index_iter_deinit (this is usually intentional, so just ignore this message) Debug: Skipping module doveadm_fts_plugin, because dlopen() failed: /usr/lib64/dovecot/doveadm/lib20_doveadm_fts_plugin.so: undefined symbol: fts_user_get_language_list (this is usually intentional, so just ignore this message) Debug: Skipping module doveadm_mail_crypt_plugin, because dlopen() failed: /usr/lib64/dovecot/doveadm/libdoveadm_mail_crypt_plugin.so: undefined symbol: mail_crypt_box_get_pvt_digests (this is usually intentional, so just ignore this message) doveadm(test4)<19830><>: Debug: auth-master: userdb lookup(test4): Started userdb lookup doveadm(test4)<19830><>: Debug: auth-master: conn unix:/var/run/dovecot/auth-userdb: Connecting doveadm(test4)<19830><>: Debug: auth-master: conn unix:/var/run/dovecot/auth-userdb (pid=15116,uid=0): Client connected (fd=10) doveadm(test4)<19830><>: Debug: auth-master: userdb lookup(test4): auth USER input: test4 system_groups_user=test4 uid=1805 gid=2300 home=/home/irams1-test/test4 doveadm(test4)<19830><>: Debug: auth-master: userdb lookup(test4): Finished userdb lookup (username=test4 system_groups_user=test4 uid=1805 gid=2300 home=/home/irams1-test/test4) doveadm(test4): Debug: Effective uid=1805, gid=2300, home=/home/irams1-test/test4 doveadm(test4): Debug: Namespace inbox: type=private, prefix=INBOX., sep=., inbox=yes, hidden=no, list=yes, subscriptions=yes location=maildir:~/Maildir doveadm(test4): Debug: maildir++: root=/home/irams1-test/test4/Maildir, index=, indexpvt=, control=, inbox=/home/irams1-test/test4/Maildir, alt= doveadm(test4): Debug: Namespace : type=private, prefix=, sep=, inbox=no, hidden=yes, list=no, subscriptions=no location=fail::LAYOUT=none doveadm(test4): Debug: none: root=, index=, indexpvt=, control=, inbox=, alt= doveadm(test4): Debug: Mailbox INBOX.spambox: Mailbox opened because: expunge doveadm(test4): Debug: expunge: box=INBOX.spambox uid=38 doveadm(test4): Debug: expunge: box=INBOX.spambox uid=39 doveadm(test4): Debug: expunge: box=INBOX.spambox uid=40 doveadm(test4): Debug: expunge: box=INBOX.spambox uid=41 doveadm(test4): Debug: expunge: box=INBOX.spambox uid=42 doveadm(test4): Debug: expunge: box=INBOX.spambox uid=43 doveadm(test4): Debug: expunge: box=INBOX.spambox uid=44 doveadm(test4): Debug: expunge: box=INBOX.spambox uid=45 doveadm(test4): Debug: expunge: box=INBOX.spambox uid=46 doveadm(test4): Debug: auth-master: conn unix:/var/run/dovecot/auth-userdb (pid=15116,uid=0): Disconnected: Connection closed (fd=10)
A few mails are deleted from the spambox but the deletion is not synced to the other side Even if I do a "doveadm force-resync -u test4 '*'" on both sides the deletions are not replicated but anyway
#doveadm replicator status test4 username priority fast sync full sync success sync failed test4 none 00:26:47 18:19:46 00:26:47 -
on both sides. If new spam arrives it got well replicated to the other side as expected
"doveconf -n" ist attached below.
Thanks, Olaf
Aki
On 04/04/2020 20:03 Olaf Hopp < olaf.hopp@kit.edu mailto:olaf.hopp@kit.edu> wrote:
Nobody ? :-(
On 3/30/20 5:26 PM, Olaf Hopp wrote:
Hello everybody, since now I did no replication and spam is delivered into users folder "spambox" Every night there is a cronjob which deletes spam older than 30 days via something like "find .... -ctime +30 -delete" Now I'm going to set up replication (two way) and I thought that doing "rm" is not a good idea. So I modified the job to something like /usr/bin/doveadm expunge -u test1 mailbox INBOX.spambox BEFORE 30d which works like intended, but I see that on the replication partner the spam isn't deleted. Even if I do a doveadm force-resync -u test1 '*' or log in via imap to the replication partner I still see the old spam.
So my question is: is this the intended behaviour and I have to run the "doveadm expunge" on both replication partners or should the deletion via "expunge" being replicated to the partner and I have a misconfiguration or maybe I hit a bug ?
Regards, Olaf
/etc/dovecot/conf.d/12-replication.conf:
mail_plugins = $mail_plugins notify replication service aggregator { fifo_listener replication-notify-fifo { mode = 0666 } unix_listener replication-notify { mode = 0666 } } service replicator { process_min_avail = 1 unix_listener replicator-doveadm { mode = 0666 } } service doveadm { inet_listener { port = 1109 } } doveadm_password = XXXXX plugin { mail_replica = tcp:X.Y.Z.X:1109 }
-- Karlsruher Institut für Technologie (KIT) ATIS - Abt. Technische Infrastruktur, Fakultät für Informatik
Dipl.-Geophys. Olaf Hopp
- Leitung IT-Dienste -
Am Fasanengarten 5, Gebäude 50.34, Raum 009 76131 Karlsruhe Telefon: +49 721 608-43973 Fax: +49 721 608-46699 E-Mail: Olaf.Hopp@kit.edu mailto:Olaf.Hopp@kit.edu www.atis.informatik.kit.edu
www.kit.edu
KIT - Die Forschungsuniversität in der Helmholtz-Gemeinschaft
Das KIT ist seit 2010 als familiengerechte Hochschule zertifiziert.
Aki Tuomi
-- Karlsruher Institut für Technologie (KIT) ATIS - Abt. Technische Infrastruktur, Fakultät für Informatik
Dipl.-Geophys. Olaf Hopp
- Leitung IT-Dienste -
Am Fasanengarten 5, Gebäude 50.34, Raum 009 76131 Karlsruhe Telefon: +49 721 608-43973 Fax: +49 721 608-46699 E-Mail: Olaf.Hopp@kit.edu www.atis.informatik.kit.edu
www.kit.edu
KIT - Die Forschungsuniversität in der Helmholtz-Gemeinschaft
Das KIT ist seit 2010 als familiengerechte Hochschule zertifiziert.
participants (2)
-
Aki Tuomi
-
Olaf Hopp