replication and spam removal ("doveadm expunge")

Olaf Hopp Olaf.Hopp at kit.edu
Mon Apr 6 13:40:19 EEST 2020


Hi Aki,

On 4/4/20 8:12 PM, Aki Tuomi wrote:
> Can you provide doveconf -n and try turning on mail_debug=yes on both ends and try doveadm -Dv expunge ....

mail_debug=yes
is on on both ends and dovecot was restarted but anyway nothing is logged when I issue "doveadm -Dv expunge "
In the shell where I issue the "expunge" I see the following:

# /usr/bin/doveadm -Dv  expunge  -u test4  mailbox INBOX.spambox  BEFORE 13m
Debug: Loading modules from directory: /usr/lib64/dovecot
Debug: Module loaded: /usr/lib64/dovecot/lib15_notify_plugin.so
Debug: Module loaded: /usr/lib64/dovecot/lib20_replication_plugin.so
Debug: Loading modules from directory: /usr/lib64/dovecot/doveadm
Debug: Skipping module doveadm_acl_plugin, because dlopen() failed: /usr/lib64/dovecot/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol: acl_user_module (this is usually intentional, so just ignore this message)
Debug: Skipping module doveadm_expire_plugin, because dlopen() failed: /usr/lib64/dovecot/doveadm/lib10_doveadm_expire_plugin.so: undefined symbol: expire_set_deinit (this is usually intentional, so just ignore this message)
Debug: Skipping module doveadm_quota_plugin, because dlopen() failed: /usr/lib64/dovecot/doveadm/lib10_doveadm_quota_plugin.so: undefined symbol: quota_user_module (this is usually intentional, so just ignore this message)
Debug: Module loaded: /usr/lib64/dovecot/doveadm/lib10_doveadm_sieve_plugin.so
Debug: Skipping module doveadm_fts_lucene_plugin, because dlopen() failed: /usr/lib64/dovecot/doveadm/lib20_doveadm_fts_lucene_plugin.so: undefined symbol: lucene_index_iter_deinit (this is usually intentional, so just ignore this message)
Debug: Skipping module doveadm_fts_plugin, because dlopen() failed: /usr/lib64/dovecot/doveadm/lib20_doveadm_fts_plugin.so: undefined symbol: fts_user_get_language_list (this is usually intentional, so just ignore this message)
Debug: Skipping module doveadm_mail_crypt_plugin, because dlopen() failed: /usr/lib64/dovecot/doveadm/libdoveadm_mail_crypt_plugin.so: undefined symbol: mail_crypt_box_get_pvt_digests (this is usually intentional, so just ignore this message)
doveadm(test4)<19830><>: Debug: auth-master: userdb lookup(test4): Started userdb lookup
doveadm(test4)<19830><>: Debug: auth-master: conn unix:/var/run/dovecot/auth-userdb: Connecting
doveadm(test4)<19830><>: Debug: auth-master: conn unix:/var/run/dovecot/auth-userdb (pid=15116,uid=0): Client connected (fd=10)
doveadm(test4)<19830><>: Debug: auth-master: userdb lookup(test4): auth USER input: test4 system_groups_user=test4 uid=1805 gid=2300 home=/home/irams1-test/test4
doveadm(test4)<19830><>: Debug: auth-master: userdb lookup(test4): Finished userdb lookup (username=test4 system_groups_user=test4 uid=1805 gid=2300 home=/home/irams1-test/test4)
doveadm(test4): Debug: Effective uid=1805, gid=2300, home=/home/irams1-test/test4
doveadm(test4): Debug: Namespace inbox: type=private, prefix=INBOX., sep=., inbox=yes, hidden=no, list=yes, subscriptions=yes location=maildir:~/Maildir
doveadm(test4): Debug: maildir++: root=/home/irams1-test/test4/Maildir, index=, indexpvt=, control=, inbox=/home/irams1-test/test4/Maildir, alt=
doveadm(test4): Debug: Namespace : type=private, prefix=, sep=, inbox=no, hidden=yes, list=no, subscriptions=no location=fail::LAYOUT=none
doveadm(test4): Debug: none: root=, index=, indexpvt=, control=, inbox=, alt=
doveadm(test4): Debug: Mailbox INBOX.spambox: Mailbox opened because: expunge
doveadm(test4): Debug: expunge: box=INBOX.spambox uid=38
doveadm(test4): Debug: expunge: box=INBOX.spambox uid=39
doveadm(test4): Debug: expunge: box=INBOX.spambox uid=40
doveadm(test4): Debug: expunge: box=INBOX.spambox uid=41
doveadm(test4): Debug: expunge: box=INBOX.spambox uid=42
doveadm(test4): Debug: expunge: box=INBOX.spambox uid=43
doveadm(test4): Debug: expunge: box=INBOX.spambox uid=44
doveadm(test4): Debug: expunge: box=INBOX.spambox uid=45
doveadm(test4): Debug: expunge: box=INBOX.spambox uid=46
doveadm(test4): Debug: auth-master: conn unix:/var/run/dovecot/auth-userdb (pid=15116,uid=0): Disconnected: Connection closed (fd=10)

A few mails are deleted from the spambox but the deletion is not synced to the other side
Even if I do a "doveadm  force-resync  -u test4 '*'" on both sides the deletions are not replicated
but anyway

#doveadm replicator status test4
username                                                                      priority fast sync full sync success sync failed
test4                                                                         none     00:26:47  18:19:46  00:26:47     -

on both sides.
If new spam arrives it got well replicated to the other side as expected

"doveconf -n" ist attached below.

Thanks,
Olaf


> 
> Aki
>> On 04/04/2020 20:03 Olaf Hopp < olaf.hopp at kit.edu <mailto:olaf.hopp at kit.edu>> wrote:
>>
>>
>> Nobody ? :-(
>>
>> On 3/30/20 5:26 PM, Olaf Hopp wrote:
>>> Hello everybody,
>>> since now I did no replication and spam is delivered into users folder "spambox"
>>> Every night there is a cronjob which deletes spam older than 30 days via something like
>>>     "find .... -ctime +30 -delete"
>>> Now I'm going to set up replication (two way) and I thought that
>>> doing "rm" is not a good idea.
>>> So I modified the job to something like
>>>     /usr/bin/doveadm expunge  -u test1 mailbox INBOX.spambox  BEFORE  30d
>>> which works like intended, but I see that on the replication
>>> partner the spam isn't deleted. Even if I do a
>>>     doveadm  force-resync  -u test1 '*'
>>> or log in via imap to the replication partner I still see the old spam.
>>>
>>> So my question is:
>>> is this the intended behaviour and I have to run the "doveadm expunge" on both replication partners
>>> or should the deletion via "expunge" being replicated to the partner and I have a misconfiguration
>>> or maybe I hit a bug ?
>>>
>>> Regards, Olaf
>>>
>>>
>>> /etc/dovecot/conf.d/12-replication.conf:
>>>
>>>
>>> mail_plugins = $mail_plugins notify replication
>>> service aggregator {
>>>         fifo_listener replication-notify-fifo {
>>>                 mode = 0666
>>>         }
>>>         unix_listener replication-notify {
>>>                 mode = 0666
>>>         }
>>> }
>>> service replicator {
>>>         process_min_avail = 1
>>>         unix_listener replicator-doveadm {
>>>                 mode = 0666
>>>         }
>>> }
>>> service doveadm {
>>>         inet_listener {
>>>                 port = 1109
>>>         }
>>> }
>>> doveadm_password = XXXXX
>>> plugin {
>>>         mail_replica = tcp:X.Y.Z.X:1109
>>> }
>>>
>>>
>>
>> -- 
>> Karlsruher Institut für Technologie (KIT)
>> ATIS - Abt. Technische Infrastruktur, Fakultät für Informatik
>>
>> Dipl.-Geophys. Olaf Hopp
>> - Leitung IT-Dienste -
>>
>> Am Fasanengarten 5, Gebäude 50.34, Raum 009
>> 76131 Karlsruhe
>> Telefon: +49 721 608-43973
>> Fax: +49 721 608-46699
>> E-Mail: Olaf.Hopp at kit.edu <mailto:Olaf.Hopp at kit.edu>
>> www.atis.informatik.kit.edu
>>
>> www.kit.edu
>>
>> KIT - Die Forschungsuniversität in der Helmholtz-Gemeinschaft
>>
>> Das KIT ist seit 2010 als familiengerechte Hochschule zertifiziert.
> 
> ---
> Aki Tuomi
> 


-- 
Karlsruher Institut für Technologie (KIT)
ATIS - Abt. Technische Infrastruktur, Fakultät für Informatik

Dipl.-Geophys. Olaf Hopp
- Leitung IT-Dienste -

Am Fasanengarten 5, Gebäude 50.34, Raum 009
76131 Karlsruhe
Telefon: +49 721 608-43973
Fax: +49 721 608-46699
E-Mail: Olaf.Hopp at kit.edu
www.atis.informatik.kit.edu

www.kit.edu

KIT - Die Forschungsuniversität in der Helmholtz-Gemeinschaft

Das KIT ist seit 2010 als familiengerechte Hochschule zertifiziert.
-------------- next part --------------
# 2.3.10 (0da0eff44): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.5.10 (bf8ef1c2)
# OS: Linux 4.18.0-147.5.1.el8_1.x86_64 x86_64 CentOS Linux release 8.1.1911 (Core)  
# Hostname: irams1-test2.ira.uka.de
auth_failure_delay = 3 secs
auth_master_user_separator = *
auth_mechanisms = plain login
auth_username_format = %Ln
auth_verbose = yes
auth_verbose_passwords = plain
auth_worker_max_count = 60
default_client_limit = 2000
default_process_limit = 3000
default_vsz_limit = 512 M
doveadm_password = # hidden, use -P to show it
first_valid_uid = 1000
last_valid_uid = 65533
mail_debug = yes
mail_location = maildir:~/Maildir
mail_plugins = " notify replication"
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext
mbox_write_locks = fcntl
namespace inbox {
  inbox = yes
  location = 
  mailbox "Deleted Items" {
    autoexpunge = 30 days
    special_use = \Trash
  }
  mailbox "Deleted Messages" {
    autoexpunge = 30 days
    special_use = \Trash
  }
  mailbox Drafts {
    auto = subscribe
    special_use = \Drafts
  }
  mailbox "Gelöschte Objekte" {
    autoexpunge = 30 days
    special_use = \Trash
  }
  mailbox "Gel&APY-schte Objekte" {
    autoexpunge = 30 days
    special_use = \Trash
  }
  mailbox Papierkorb {
    autoexpunge = 30 days
    special_use = \Trash
  }
  mailbox Sent {
    auto = subscribe
    special_use = \Sent
  }
  mailbox Trash {
    auto = subscribe
    autoexpunge = 30 days
    special_use = \Trash
  }
  mailbox spambox {
    auto = create
    special_use = \Junk
  }
  prefix = INBOX.
  separator = .
}
passdb {
  args = /etc/dovecot/master-users
  driver = passwd-file
  master = yes
}
passdb {
  args = dovecot
  driver = pam
}
plugin {
  mail_replica = tcp:141.3.10.15:1109
  sieve = file:~/sieve;active=~/.dovecot.sieve
  sieve_before = /etc/dovecot/sieve-master
  sieve_max_redirects = 20
}
postmaster_address = postmaster at ira.uka.de
protocols = imap pop3 lmtp sieve sieve
quota_full_tempfail = yes
service aggregator {
  fifo_listener replication-notify-fifo {
    mode = 0666
  }
  unix_listener replication-notify {
    mode = 0666
  }
}
service doveadm {
  inet_listener {
    port = 1109
  }
}
service imap-login {
  process_limit = 8192
  process_min_avail = 16
  service_count = 0
}
service imap {
  process_limit = 8192
}
service lmtp {
  executable = lmtp -L
}
service managesieve-login {
  inet_listener sieve {
    port = 4190
  }
}
service pop3 {
  process_limit = 8192
}
service replicator {
  process_min_avail = 1
  unix_listener replicator-doveadm {
    mode = 0666
  }
}
ssl = required
ssl_ca = </etc/pki/tls/certs/kitchain.crt
ssl_cert = </etc/pki/tls/certs/imap.informatik.pem
ssl_cipher_list = ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA
ssl_dh = # hidden, use -P to show it
ssl_key = # hidden, use -P to show it
ssl_prefer_server_ciphers = yes
syslog_facility = local3
userdb {
  driver = passwd
}
verbose_proctitle = yes
protocol lmtp {
  mail_plugins = " notify replication sieve"
}
protocol imap {
  mail_max_userip_connections = 100
  ssl_cert = </etc/pki/tls/certs/imap_with_chain.informatik.kit.edu.pem
  ssl_key = # hidden, use -P to show it
}
protocol pop3 {
  ssl_cert = </etc/pki/tls/certs/pop_with_chain.informatik.kit.edu.pem
  ssl_key = # hidden, use -P to show it
}
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5212 bytes
Desc: S/MIME Cryptographic Signature
URL: <https://dovecot.org/pipermail/dovecot/attachments/20200406/14430fdf/attachment-0001.p7s>


More information about the dovecot mailing list