doveadm expunge didn't clear Trash mailbox
Hi,
Why did my 'doveadm expunge' command not clear the Trash out of a mailbox?
The oldest messages was put into the Trash on the 26th June 2018 as shown in the file timestamp.
-rw------- 1 vmail mail 33056 Jun 26 16:46 1530031582.M768773P18242.mx10,S=33056,W=33510:2,RS
# doveadm -D expunge -u testuser@example.org mailbox Trash savedbefore 2d Debug: Loading modules from directory: /usr/lib/dovecot/modules/doveadm Debug: Skipping module doveadm_acl_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol: acl_lookup_dict_iterate_visible_next (this is usually intentional, so just ignore this message) Debug: Skipping module doveadm_expire_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_expire_plugin.so: undefined symbol: expire_set_deinit (this is usually intentional, so just ignore this message) Debug: Skipping module doveadm_quota_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_quota_plugin.so: undefined symbol: quota_user_module (this is usually intentional, so just ignore this message) Debug: Module loaded: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_sieve_plugin.so Debug: Skipping module doveadm_fts_lucene_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_lucene_plugin.so: undefined symbol: lucene_index_iter_deinit (this is usually intentional, so just ignore this message) Debug: Skipping module doveadm_fts_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_plugin.so: undefined symbol: fts_user_get_language_list (this is usually intentional, so just ignore this message) Debug: Skipping module doveadm_mail_crypt_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/libdoveadm_mail_crypt_plugin.so: undefined symbol: mail_crypt_box_get_pvt_digests (this is usually intentional, so just ignore this message) doveadm(testuser@example.org): Debug: Added userdb setting: mail=maildir:/var/vmail/example.org/testuser@example.org doveadm(testuser@example.org): Debug: Added userdb setting: plugin/quota=dirsize:storage= doveadm(testuser@example.org): Debug: Effective uid=150, gid=8, home=/var/vmail/example.org/testuser@example.org doveadm(testuser@example.org): Debug: Namespace inbox: type=private, prefix=, sep=, inbox=yes, hidden=no, list=yes, subscriptions=yes location=maildir:/var/vmail/example.org/testuser@example.org doveadm(testuser@example.org): Debug: maildir++: root=/var/vmail/example.org/testuser@example.org, index=, indexpvt=, control=, inbox=/var/vmail/example.org/testuser@example.org, alt=
Sophie
On Wed, 18 Jul 2018, Sophie Loewenthal wrote:
Why did my 'doveadm expunge' command not clear the Trash out of a mailbox?
The oldest messages was put into the Trash on the 26th June 2018 as shown in the file timestamp.
-rw------- 1 vmail mail 33056 Jun 26 16:46 1530031582.M768773P18242.mx10,S=33056,W=33510:2,RS
# doveadm -D expunge -u testuser@example.org mailbox Trash savedbefore 2d
I was going to suggest that you dump the savedbefore dates on all your messages to see what Dovecot thinks your message's timestamps are. I tried it out myself and was surprised to find *all* messages in *any* mailbox I looked at gave the same timestamp -- probably that of the latest message.
For example,
# doveadm -f tab fetch -u {user} "uid date.saved" mailbox Trash
uid date.saved
23624 2018-07-18 14:50:03
23625 2018-07-18 14:50:03
[... 3657 lines removed ...]
27295 2018-07-18 14:50:03
27296 2018-07-18 14:50:03
27297 2018-07-18 14:50:03
# doveadm -f tab fetch -u {user} "uid date.saved" mailbox INBOX
uid date.saved
94878 2018-07-18 14:53:46
100366 2018-07-18 14:53:46
[... 140 lines removed ...]
102313 2018-07-18 14:53:46
102314 2018-07-18 14:53:46
102315 2018-07-18 14:53:46This would explain why the cron script I use to blow away old deleted mail has not fired in ages.
Is this a bug, or am I not understanding what "savedbefore" means now? It certainly has changed behaviour since <2.2, since my script used to work before.
Joseph Tam jtam.home@gmail.com
And here is mine. All dates are todays!
# doveadm -f tab fetch -u test@example.org "uid date.saved" mailbox Trash uid date.saved 6829 2018-07-18 06:04:50 6830 2018-07-18 11:10:13 6831 2018-07-18 16:16:15 6832 2018-07-18 16:16:15 6833 2018-07-18 08:22:55 6834 2018-07-18 16:16:15 6835 2018-07-18 08:47:10 6836 2018-07-18 09:12:11 6837 2018-07-18 09:53:31 6838 2018-07-18 11:03:31 SNIP a lot of lines... 6873 2018-07-18 18:54:07 6874 2018-07-18 18:54:07 6875 2018-07-18 18:54:07 6876 2018-07-18 18:54:07 6877 2018-07-18 21:23:08
# doveadm -f tab fetch -u sophie@klunky.org "uid date.saved" mailbox Inbox|head uid date.saved 57 2018-01-23 21:05:06 60 2018-01-23 21:05:06 61 2018-01-23 21:05:06 62 2018-01-23 21:05:06 64 2018-01-23 21:05:06 SNIP a lot of lines 7734 2018-07-02 20:48:37 7780 2018-07-06 18:50:37 7794 2018-07-09 08:03:45 7833 2018-07-11 15:27:06
This is on Debian 9.5 with dovecot 1:2.2.27-3+deb9u2
On 19 Jul 2018, at 00:05, Joseph Tam jtam.home@gmail.com wrote:
On Wed, 18 Jul 2018, Sophie Loewenthal wrote:
Why did my 'doveadm expunge' command not clear the Trash out of a mailbox?
The oldest messages was put into the Trash on the 26th June 2018 as shown in the file timestamp.
-rw------- 1 vmail mail 33056 Jun 26 16:46 1530031582.M768773P18242.mx10,S=33056,W=33510:2,RS
# doveadm -D expunge -u testuser@example.org mailbox Trash savedbefore 2d
I was going to suggest that you dump the savedbefore dates on all your messages to see what Dovecot thinks your message's timestamps are. I tried it out myself and was surprised to find *all* messages in *any* mailbox I looked at gave the same timestamp -- probably that of the latest message.
For example,
# doveadm -f tab fetch -u {user} "uid date.saved" mailbox Trash uid date.saved 23624 2018-07-18 14:50:03 23625 2018-07-18 14:50:03 [... 3657 lines removed ...] 27295 2018-07-18 14:50:03 27296 2018-07-18 14:50:03 27297 2018-07-18 14:50:03
# doveadm -f tab fetch -u {user} "uid date.saved" mailbox INBOX uid date.saved 94878 2018-07-18 14:53:46 100366 2018-07-18 14:53:46 [... 140 lines removed ...] 102313 2018-07-18 14:53:46 102314 2018-07-18 14:53:46 102315 2018-07-18 14:53:46
This would explain why the cron script I use to blow away old deleted mail has not fired in ages.
Is this a bug, or am I not understanding what "savedbefore" means now? It certainly has changed behaviour since <2.2, since my script used to work before.
Joseph Tam jtam.home@gmail.com
Sorry, mistake:
I meant all dates in Trash are dated today, but the times differ.
Inbox is different, but the last file in user@domain/cur differs from dovecot fetch date.saved.
-rw------- 1 vmail mail 3566 2011-10-08 13:06:16.000000000 +0000 1371065422.M299803P23725.logout,S=3566,W=3659:2,Shj -rw------- 1 vmail mail 14621 2011-09-16 11:49:45.000000000 +0000 1507747187.M109389P30107.logout,W=14988,S=14621:2,Shj
On 19 Jul 2018, at 00:17, Sophie Loewenthal sophie@klunky.co.uk wrote:
And here is mine. All dates are todays!
# doveadm -f tab fetch -u test@example.org "uid date.saved" mailbox Trash uid date.saved 6829 2018-07-18 06:04:50 6830 2018-07-18 11:10:13 6831 2018-07-18 16:16:15 6832 2018-07-18 16:16:15 6833 2018-07-18 08:22:55 6834 2018-07-18 16:16:15 6835 2018-07-18 08:47:10 6836 2018-07-18 09:12:11 6837 2018-07-18 09:53:31 6838 2018-07-18 11:03:31 SNIP a lot of lines... 6873 2018-07-18 18:54:07 6874 2018-07-18 18:54:07 6875 2018-07-18 18:54:07 6876 2018-07-18 18:54:07 6877 2018-07-18 21:23:08
# doveadm -f tab fetch -u sophie@klunky.org "uid date.saved" mailbox Inbox|head uid date.saved 57 2018-01-23 21:05:06 60 2018-01-23 21:05:06 61 2018-01-23 21:05:06 62 2018-01-23 21:05:06 64 2018-01-23 21:05:06 SNIP a lot of lines 7734 2018-07-02 20:48:37 7780 2018-07-06 18:50:37 7794 2018-07-09 08:03:45 7833 2018-07-11 15:27:06
This is on Debian 9.5 with dovecot 1:2.2.27-3+deb9u2
On 19 Jul 2018, at 00:05, Joseph Tam jtam.home@gmail.com wrote:
On Wed, 18 Jul 2018, Sophie Loewenthal wrote:
Why did my 'doveadm expunge' command not clear the Trash out of a mailbox?
The oldest messages was put into the Trash on the 26th June 2018 as shown in the file timestamp.
-rw------- 1 vmail mail 33056 Jun 26 16:46 1530031582.M768773P18242.mx10,S=33056,W=33510:2,RS
# doveadm -D expunge -u testuser@example.org mailbox Trash savedbefore 2d
I was going to suggest that you dump the savedbefore dates on all your messages to see what Dovecot thinks your message's timestamps are. I tried it out myself and was surprised to find *all* messages in *any* mailbox I looked at gave the same timestamp -- probably that of the latest message.
For example,
# doveadm -f tab fetch -u {user} "uid date.saved" mailbox Trash uid date.saved 23624 2018-07-18 14:50:03 23625 2018-07-18 14:50:03 [... 3657 lines removed ...] 27295 2018-07-18 14:50:03 27296 2018-07-18 14:50:03 27297 2018-07-18 14:50:03
# doveadm -f tab fetch -u {user} "uid date.saved" mailbox INBOX uid date.saved 94878 2018-07-18 14:53:46 100366 2018-07-18 14:53:46 [... 140 lines removed ...] 102313 2018-07-18 14:53:46 102314 2018-07-18 14:53:46 102315 2018-07-18 14:53:46
This would explain why the cron script I use to blow away old deleted mail has not fired in ages.
Is this a bug, or am I not understanding what "savedbefore" means now? It certainly has changed behaviour since <2.2, since my script used to work before.
Joseph Tam jtam.home@gmail.com
I wrote:
I was going to suggest that you dump the savedbefore dates on all your messages to see what Dovecot thinks your message's timestamps are. I tried it out myself and was surprised to find *all* messages in *any* mailbox I looked at gave the same timestamp -- probably that of the latest message.
For example,
# doveadm -f tab fetch -u {user} "uid date.saved" mailbox Trash uid date.saved 23624 2018-07-18 14:50:03 23625 2018-07-18 14:50:03 [... 3657 lines removed ...] 27295 2018-07-18 14:50:03 27296 2018-07-18 14:50:03 27297 2018-07-18 14:50:03
I wrote about a related issue several years back, starting with
https://dovecot.org/list/dovecot/2012-March/134706.htmlI'm not sure how I got over this and got usable "date.saved" cache values, and why it stopped working. Maybe I need to set something like
mail_always_cache_fields = date.saved
mail_cache_fields = date.savedto force date.saved to be cached.
On Thu, 19 Jul 2018, Sophie Loewenthal wrote:
And here is mine. All dates are todays!
# doveadm -f tab fetch -u test@example.org "uid date.saved" mailbox Trash uid date.saved 6829 2018-07-18 06:04:50 6830 2018-07-18 11:10:13 6831 2018-07-18 16:16:15 SNIP a lot of lines... 6873 2018-07-18 18:54:07 6874 2018-07-18 18:54:07 6875 2018-07-18 18:54:07 6876 2018-07-18 18:54:07 6877 2018-07-18 21:23:08
You may or may not have the same problem I do -- your timestamps actually differ, whereas mine are all the same.
Joseph Tam jtam.home@gmail.com
On Wed, 18 Jul 2018, Joseph Tam wrote:
I wrote about a related issue several years back, starting with
https://dovecot.org/list/dovecot/2012-March/134706.html
I'm not sure how I got over this and got usable "date.saved" cache values, and why it stopped working. Maybe I need to set something like
mail_always_cache_fields = date.saved mail_cache_fields = date.saved
to force date.saved to be cached.
This doesn't work -- it's not a cachable value.
To get around the wonky date.saved values, I used date.received instead, which seems to have more reliable values
doveadm -f tab fetch -u {user} "uid date.received" mailbox {box}and the corresponding expunge command would be
doveadm expunge -u {user} mailbox {box} BEFORE {time-limit}It not quite the same as it's not the time it was copied into {box} but rather, the internal date, which may be earlier (i.e when it was received).
Joseph Tam jtam.home@gmail.com
On Jul 18, 2018 um 15:05:46, Joseph Tam wrote:
On Wed, 18 Jul 2018, Sophie Loewenthal wrote:
Why did my 'doveadm expunge' command not clear the Trash out of a mailbox?
The oldest messages was put into the Trash on the 26th June 2018 as shown in the file timestamp.
-rw------- 1 vmail mail 33056 Jun 26 16:46 1530031582.M768773P18242.mx10,S=33056,W=33510:2,RS
# doveadm -D expunge -u testuser@example.org mailbox Trash savedbefore 2d
I was going to suggest that you dump the savedbefore dates on all your messages to see what Dovecot thinks your message's timestamps are. I tried it out myself and was surprised to find *all* messages in *any* mailbox I looked at gave the same timestamp -- probably that of the latest message.
For example,
# doveadm -f tab fetch -u {user} "uid date.saved" mailbox Trash uid date.saved 23624 2018-07-18 14:50:03 23625 2018-07-18 14:50:03 [... 3657 lines removed ...] 27295 2018-07-18 14:50:03 27296 2018-07-18 14:50:03 27297 2018-07-18 14:50:03
# doveadm -f tab fetch -u {user} "uid date.saved" mailbox INBOX uid date.saved 94878 2018-07-18 14:53:46 100366 2018-07-18 14:53:46 [... 140 lines removed ...] 102313 2018-07-18 14:53:46 102314 2018-07-18 14:53:46 102315 2018-07-18 14:53:46
This would explain why the cron script I use to blow away old deleted mail has not fired in ages.
Is this a bug, or am I not understanding what "savedbefore" means now? It certainly has changed behaviour since <2.2, since my script used to work before.
Hello Joseph,
here works a dovecot 2.2.27 on a raspberrypi and the behaviour is as expected.
doveadm -f tab fetch -u <user> "uid date.saved" mailbox Trash uid date.saved 314 2018-06-23 00:35:59 315 2018-06-23 12:39:10 316 2018-06-24 10:32:43 317 2018-06-24 10:32:43 318 2018-06-24 10:32:47 319 2018-06-24 14:26:24 320 2018-06-24 14:26:24 321 2018-06-24 14:28:45 322 2018-06-24 14:28:45 -- lines skipped 574 2018-07-20 20:58:21 575 2018-07-21 17:22:17 576 2018-07-22 15:15:18 577 2018-07-22 15:28:28 578 2018-07-22 15:28:28 579 2018-07-22 15:28:28 580 2018-07-22 16:12:47 581 2018-07-23 13:27:12 582 2018-07-23 12:55:26 583 2018-07-23 13:19:01 584 2018-07-23 13:19:01 585 2018-07-23 13:49:01 586 2018-07-23 13:52:46
And I have a cron script that expunges the mails older than 30 days.
/usr/bin/doveadm expunge -u <user> mailbox Trash savedbefore 30d
Michael
participants (3)
-
Joseph Tam
-
Michael Wagner
-
Sophie Loewenthal