[Dovecot] Trouble with expire plugin
Hi all,
Maybe someone here can help me with this problem, not sure if this is a configuration issue or an actual bug.
my daily cron job says:
#!/bin/sh doveadm expunge -A mailbox Junk savedbefore 14d doveadm expunge -A mailbox Trash savedbefore 30d doveadm expunge -A mailbox "Trash.*" savedbefore 30d
But it hasn't been deleting any messages in a long time, I'm not sure when it stopped working.
I've also noticed that it does delete matching rows in the expires database (expire_stamp < savedbefore days). Furthermore, rows are deleted even when running doveadm search, and regardless of the mailbox column, i.e. even when != the mailbox name being searched! Anyway why should doveadm search delete rows in the expires database?
The database is mysql.
---- dovecot -n ---- # 2.1.7: /etc/dovecot/dovecot.conf # OS: Linux 3.2.0-0.bpo.3-amd64 x86_64 Debian 6.0.6 auth_default_realm = xxxxxxx.com auth_mechanisms = plain login cram-md5 auth_verbose = yes auth_verbose_passwords = plain default_client_limit = 8096 dict { expire = mysql:/etc/dovecot/dovecot-dict-sql.conf.ext quota = mysql:/etc/dovecot/dovecot-dict-sql.conf.ext } disable_plaintext_auth = no mail_gid = vmail mail_plugins = quota expire listescape mail_uid = vmail managesieve_sieve_capability = fileinto envelope encoded-character subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables mailbox date ihave namespace { inbox = yes location = prefix = separator = / type = private } passdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql } plugin { antispam_backend = pipe antispam_pipe_program = /usr/bin/ssh antispam_pipe_program_args = -l;spamd;-i;/etc/dovecot/sareport.key;mx1.xxxxxxx.com antispam_pipe_program_notspam_arg = revoke antispam_pipe_program_spam_arg = report antispam_pipe_tmpdir = /tmp antispam_spam = Junk antispam_trash_pattern_ignorecase = trash;Deleted * autocreate = Trash autocreate2 = Junk autocreate3 = Drafts autocreate4 = Sent autosubscribe = Trash autosubscribe2 = Junk autosubscribe3 = Drafts autosubscribe4 = Sent expire = Trash expire2 = Trash.* expire3 = Junk expire_dict = proxy::expire quota = dict:user::proxy::quota quota_rule = *:storage=2G quota_rule2 = Trash:storage=+250M quota_rule3 = Junk:ignore quota_warning = storage=99%% quota-warning 99 %u quota_warning2 = storage=95%% quota-warning 95 %u sieve = ~/.dovecot.sieve sieve_before = /var/local/lib/dovecot/sieve/global/fileinto-spam.sieve sieve_dir = ~/sieve sieve_extensions = -vacation -body -reject -enotify -environment -virustest -spamtest sieve_max_actions = 32 sieve_max_redirects = 4 sieve_max_script_size = 10K sieve_quota_max_scripts = 10 } postmaster_address = postmaster@xxxxxxx.com protocols = imap pop3 lmtp sieve service auth-worker { user = $default_internal_user } service auth { client_limit = 20480 unix_listener auth-userdb { group = root mode = 0600 user = vmail } } service dict { unix_listener dict { mode = 0600 user = vmail } } service imap-login { process_min_avail = 4 service_count = 0 vsz_limit = 128 M } service imap { process_limit = 10240 } service lmtp { inet_listener lmtp { address = 0.0.0.0 port = 7025 } } service managesieve-login { inet_listener sieve { port = 4190 } service_count = 0 vsz_limit = 128 M } service pop3-login { process_min_avail = 4 service_count = 0 vsz_limit = 128 M } service pop3 { process_limit = 4096 } service quota-warning { executable = script /usr/local/bin/quota-warning unix_listener quota-warning { user = vmail } user = vmail } ssl_cert =
in dovecot-sql.conf.ext I have: iterate_query = SELECT userid as user, domain FROM email where deleted=0
---- dovecot-dict-sql.conf.ext ---- connect = host=localhost dbname=email user=email map { pattern = priv/quota/storage table = quota username_field = username value_field = bytes } map { pattern = priv/quota/messages table = quota username_field = username value_field = messages } map { pattern = shared/expire/$user/$mailbox table = expires value_field = expire_stamp fields { username = $user mailbox = $mailbox } }
Anyone..? Is there any further detail I can provide? This is really puzzling me.
Gedalya
On 01/02/2013 06:44 PM, Gedalya wrote:
Hi all,
Maybe someone here can help me with this problem, not sure if this is a configuration issue or an actual bug.
my daily cron job says:
#!/bin/sh doveadm expunge -A mailbox Junk savedbefore 14d doveadm expunge -A mailbox Trash savedbefore 30d doveadm expunge -A mailbox "Trash.*" savedbefore 30d
But it hasn't been deleting any messages in a long time, I'm not sure when it stopped working.
I've also noticed that it does delete matching rows in the expires database (expire_stamp < savedbefore days). Furthermore, rows are deleted even when running doveadm search, and regardless of the mailbox column, i.e. even when != the mailbox name being searched! Anyway why should doveadm search delete rows in the expires database?
The database is mysql.
---- dovecot -n ---- # 2.1.7: /etc/dovecot/dovecot.conf # OS: Linux 3.2.0-0.bpo.3-amd64 x86_64 Debian 6.0.6 auth_default_realm = xxxxxxx.com auth_mechanisms = plain login cram-md5 auth_verbose = yes auth_verbose_passwords = plain default_client_limit = 8096 dict { expire = mysql:/etc/dovecot/dovecot-dict-sql.conf.ext quota = mysql:/etc/dovecot/dovecot-dict-sql.conf.ext } disable_plaintext_auth = no mail_gid = vmail mail_plugins = quota expire listescape mail_uid = vmail managesieve_sieve_capability = fileinto envelope encoded-character subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables mailbox date ihave namespace { inbox = yes location = prefix = separator = / type = private } passdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql } plugin { antispam_backend = pipe antispam_pipe_program = /usr/bin/ssh antispam_pipe_program_args = -l;spamd;-i;/etc/dovecot/sareport.key;mx1.xxxxxxx.com antispam_pipe_program_notspam_arg = revoke antispam_pipe_program_spam_arg = report antispam_pipe_tmpdir = /tmp antispam_spam = Junk antispam_trash_pattern_ignorecase = trash;Deleted * autocreate = Trash autocreate2 = Junk autocreate3 = Drafts autocreate4 = Sent autosubscribe = Trash autosubscribe2 = Junk autosubscribe3 = Drafts autosubscribe4 = Sent expire = Trash expire2 = Trash.* expire3 = Junk expire_dict = proxy::expire quota = dict:user::proxy::quota quota_rule = *:storage=2G quota_rule2 = Trash:storage=+250M quota_rule3 = Junk:ignore quota_warning = storage=99%% quota-warning 99 %u quota_warning2 = storage=95%% quota-warning 95 %u sieve = ~/.dovecot.sieve sieve_before = /var/local/lib/dovecot/sieve/global/fileinto-spam.sieve sieve_dir = ~/sieve sieve_extensions = -vacation -body -reject -enotify -environment -virustest -spamtest sieve_max_actions = 32 sieve_max_redirects = 4 sieve_max_script_size = 10K sieve_quota_max_scripts = 10 } postmaster_address = postmaster@xxxxxxx.com protocols = imap pop3 lmtp sieve service auth-worker { user = $default_internal_user } service auth { client_limit = 20480 unix_listener auth-userdb { group = root mode = 0600 user = vmail } } service dict { unix_listener dict { mode = 0600 user = vmail } } service imap-login { process_min_avail = 4 service_count = 0 vsz_limit = 128 M } service imap { process_limit = 10240 } service lmtp { inet_listener lmtp { address = 0.0.0.0 port = 7025 } } service managesieve-login { inet_listener sieve { port = 4190 } service_count = 0 vsz_limit = 128 M } service pop3-login { process_min_avail = 4 service_count = 0 vsz_limit = 128 M } service pop3 { process_limit = 4096 } service quota-warning { executable = script /usr/local/bin/quota-warning unix_listener quota-warning { user = vmail } user = vmail } ssl_cert =
in dovecot-sql.conf.ext I have: iterate_query = SELECT userid as user, domain FROM email where deleted=0
---- dovecot-dict-sql.conf.ext ---- connect = host=localhost dbname=email user=email map { pattern = priv/quota/storage table = quota username_field = username value_field = bytes } map { pattern = priv/quota/messages table = quota username_field = username value_field = messages } map { pattern = shared/expire/$user/$mailbox table = expires value_field = expire_stamp fields { username = $user mailbox = $mailbox } }
Am 09.01.2013 03:23, schrieb Gedalya:
Anyone..? Is there any further detail I can provide? This is really puzzling me.
Gedalya
On 01/02/2013 06:44 PM, Gedalya wrote:
Hi all,
Maybe someone here can help me with this problem, not sure if this is a configuration issue or an actual bug.
my daily cron job says:
#!/bin/sh doveadm expunge -A mailbox Junk savedbefore 14d doveadm expunge -A mailbox Trash savedbefore 30d doveadm expunge -A mailbox "Trash.*" savedbefore 30d
do verbose logging during cron
Best Regards MfG Robert Schetterer
-- [*] sys4 AG
http://sys4.de, +49 (89) 30 90 46 64 Franziskanerstraße 15, 81669 München
Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263 Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer Aufsichtsratsvorsitzender: Joerg Heidrich
On 01/09/2013 02:32 AM, Robert Schetterer wrote:
my daily cron job says:
#!/bin/sh doveadm expunge -A mailbox Junk savedbefore 14d doveadm expunge -A mailbox Trash savedbefore 30d doveadm expunge -A mailbox "Trash.*" savedbefore 30d
do verbose logging during cron
Best Regards MfG Robert Schetterer No need, I have indeed already gone beyond the "but I made a cron job, why isn't it working" point.
mysql> select * from expires; . . . 7 rows in set (0.00 sec)
Complete doveadm output with domain masked:
# doveadm -D search -A mailbox Trash savedbefore 14d doveadm(root): Debug: Loading modules from directory: /usr/lib/dovecot/modules doveadm(root): Debug: Module loaded: /usr/lib/dovecot/modules/lib10_quota_plugin.so doveadm(root): Debug: Module loaded: /usr/lib/dovecot/modules/lib20_expire_plugin.so doveadm(root): Debug: Module loaded: /usr/lib/dovecot/modules/lib20_listescape_plugin.so doveadm(root): Debug: Loading modules from directory: /usr/lib/dovecot/modules/doveadm doveadm(root): Debug: Skipping module doveadm_acl_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol: acl_user_module (this is usually intentional, so just ignore this message) doveadm(root): Debug: Module loaded: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_expire_plugin.so doveadm(root): Debug: Module loaded: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_quota_plugin.so doveadm(root): Debug: Skipping module doveadm_zlib_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_zlib_plugin.so: undefined symbol: i_stream_create_deflate (this is usually intentional, so just ignore this message) doveadm(root): Debug: Skipping module doveadm_fts_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_plugin.so: undefined symbol: fts_list_backend (this is usually intentional, so just ignore this message) doveadm(root): Debug: expire: Searching only users listed in expire database doveadm(root): Debug: expire: Stopping iteration on key shared/expire/jedi1@xxxxxx.com/Trash (1357165578 > 1356528524)
mysql> select * from expires; . . . 6 rows in set (0.00 sec)
The row being deleted: levy@xxxxxxx.com, Junk, 1350397036 (Tue Oct 16 10:17:16 EDT 2012), all others at this point in time are younger than 14 days. So far it has seemed to me that running any search or expunge has the sole effect of deleting rows matching the savedbefore criteria, regardless of the mailbox criteria.
working with doveadm -o"mail_plugins=....." leaving out expire seems to work fine.
On 01/09/2013 08:37 AM, Gedalya wrote:
On 01/09/2013 02:32 AM, Robert Schetterer wrote:
my daily cron job says:
#!/bin/sh doveadm expunge -A mailbox Junk savedbefore 14d doveadm expunge -A mailbox Trash savedbefore 30d doveadm expunge -A mailbox "Trash.*" savedbefore 30d
do verbose logging during cron
Best Regards MfG Robert Schetterer No need, I have indeed already gone beyond the "but I made a cron job, why isn't it working" point.
mysql> select * from expires; . . . 7 rows in set (0.00 sec)
Complete doveadm output with domain masked:
# doveadm -D search -A mailbox Trash savedbefore 14d doveadm(root): Debug: Loading modules from directory: /usr/lib/dovecot/modules doveadm(root): Debug: Module loaded: /usr/lib/dovecot/modules/lib10_quota_plugin.so doveadm(root): Debug: Module loaded: /usr/lib/dovecot/modules/lib20_expire_plugin.so doveadm(root): Debug: Module loaded: /usr/lib/dovecot/modules/lib20_listescape_plugin.so doveadm(root): Debug: Loading modules from directory: /usr/lib/dovecot/modules/doveadm doveadm(root): Debug: Skipping module doveadm_acl_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol: acl_user_module (this is usually intentional, so just ignore this message) doveadm(root): Debug: Module loaded: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_expire_plugin.so doveadm(root): Debug: Module loaded: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_quota_plugin.so doveadm(root): Debug: Skipping module doveadm_zlib_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_zlib_plugin.so: undefined symbol: i_stream_create_deflate (this is usually intentional, so just ignore this message) doveadm(root): Debug: Skipping module doveadm_fts_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_plugin.so: undefined symbol: fts_list_backend (this is usually intentional, so just ignore this message) doveadm(root): Debug: expire: Searching only users listed in expire database doveadm(root): Debug: expire: Stopping iteration on key shared/expire/jedi1@xxxxxx.com/Trash (1357165578 > 1356528524)
mysql> select * from expires; . . . 6 rows in set (0.00 sec)
The row being deleted: levy@xxxxxxx.com, Junk, 1350397036 (Tue Oct 16 10:17:16 EDT 2012), all others at this point in time are younger than 14 days. So far it has seemed to me that running any search or expunge has the sole effect of deleting rows matching the savedbefore criteria, regardless of the mailbox criteria.
working with doveadm -o"mail_plugins=....." leaving out expire seems to work fine.
I got it working. I know this worked before, I can't specify when it stopped working, but here is the story.
I have: auth_default_realm = domain.com
in dovecot-sql.conf.ext I had: iterate_query = SELECT userid as user, domain FROM email where deleted=0
Now it's working when I changed it to: iterate_query = SELECT concat(userid, '@', domain) as username FROM email where deleted=0
I noticed this when running doveadm expunge (without expire plugin) and saw lines like the following: doveadm(jeremy): Info: User no longer exists, skipping
This user exists, but on another domain, not the default one.
Users under the default domain come up as normal: doveadm(hr): Debug: Added userdb setting: mail=maildir:/stor/mail/domains/domain.com/hr/hr/Maildir
After changing iterate query all users come up as fully qualified, like doveadm(jeremy@otherdomain.com): Debug: Added userdb setting: mail=maildir:/stor/mail/domains/otherdomain.com/je/jeremy/Maildir
So, with regards to the expires issue, I guess the reason why the expires table wasn't working is because it contains the full email address in the username field and doveadm here was working with the userid alone.
Is this a bug? The wiki clearly says that iterate_query can take the domain in a separate field, yet dovecot seems to ignore the domain field and use auth_default_realm.
Thanks,
Gedalya
On 27.1.2013, at 22.49, Gedalya gedalya@gedalya.net wrote:
I got it working. I know this worked before, I can't specify when it stopped working, but here is the story.
I have: auth_default_realm = domain.com
in dovecot-sql.conf.ext I had: iterate_query = SELECT userid as user, domain FROM email where deleted=0
Now it's working when I changed it to: iterate_query = SELECT concat(userid, '@', domain) as username FROM email where deleted=0
iterate_query = SELECT userid as username, domain FROM email where deleted=0
would have worked too. user = the whole user@domain, while username = without domain. I think I'll leave it like this.
Is this a bug? The wiki clearly says that iterate_query can take the domain in a separate field, yet dovecot seems to ignore the domain field and use auth_default_realm.
The wiki page was partially wrong though, updated.
participants (3)
-
Gedalya
-
Robert Schetterer
-
Timo Sirainen