Autoexpunge not working for Junk?

newer
Dovecot not responding to external...

Amir Caspi

25 Jul 2019 25 Jul '19

7:18 a.m.

Hi all,

I set up dovecot a couple of months ago and am having trouble getting autoexpunge=30d to work on my Trash and Junk mailboxes.  Not sure why not because I'm not getting error messages in my log.
Running "doveadm search -u <user> mailbox Junk savedbefore 30d" shows me many messages (I've got messages back to mid-May, and a couple of other users have them back to early April, although if this setting were working, there should be nothing earlier than June 24).  Running a manual doveadm expunge works fine... it's just autoexpunge that seems to not be running at all.

I'm using sendmail as the MTA and procmail as the LDA, so dovecot is running purely for IMAP/POP service.

Any help is much appreciated.

Thanks!

doveconf -n: # 2.2.36 (1f10bfa63): /etc/dovecot/dovecot.conf # OS: Linux 3.10.0-957.21.3.el7.x86_64 x86_64 CentOS Linux release 7.6.1810 (Core)
# Hostname: REDACTED auth_username_format = %Ln first_valid_uid = 1000 mail_location = mbox:~/mail:INBOX=/var/spool/mail/%u mailbox_list_index = yes mbox_write_locks = fcntl namespace compat1 { alias_for = hidden = yes list = no location = prefix = mail/ separator = / } namespace compat2 { alias_for = hidden = yes list = no location = prefix = ~/mail/ separator = / } namespace compat3 { alias_for = hidden = yes list = no location = prefix = ~%u/mail/ separator = / } namespace inbox { inbox = yes location = mailbox Archive { special_use = \Archive } mailbox "Deleted Messages" { autoexpunge = 30 days special_use = \Trash } mailbox Drafts { special_use = \Drafts } mailbox Junk { autoexpunge = 30 days special_use = \Junk } mailbox "Junk E-mail" { autoexpunge = 30 days special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Spam { autoexpunge = 30 days special_use = \Junk } mailbox Trash { autoexpunge = 30 days special_use = \Trash } prefix = separator = / } passdb { driver = pam } pop3_uidl_format = %08Xv%08Xu ssl_cert = # REDACTED ssl_cipher_list = # REDACTED ssl_dh_parameters_length = # REDACTED ssl_key = # hidden, use -P to show it ssl_prefer_server_ciphers = yes userdb { driver = passwd }

--- Amir

Show replies by date

Timo Sirainen

27 Jul 27 Jul

2:23 p.m.

On 25 Jul 2019, at 7.18, Amir Caspi via dovecot dovecot@dovecot.org wrote:

...

Hi all,

I set up dovecot a couple of months ago and am having trouble getting autoexpunge=30d to work on my Trash and Junk mailboxes. Not sure why not because I'm not getting error messages in my log. Running "doveadm search -u <user> mailbox Junk savedbefore 30d" shows me many messages (I've got messages back to mid-May, and a couple of other users have them back to early April, although if this setting were working, there should be nothing earlier than June 24). Running a manual doveadm expunge works fine... it's just autoexpunge that seems to not be running at all.

Autoexpunging tries to be efficient, so it looks only at the first email's saved-timestamp. It's also cached in dovecot.list.index. So you should check:

What's the first mail's saved-timestamp? doveadm fetch -u user date.saved mailbox Junk 1
That timestamp should also be the same in dovecot.list.index: doveadm mailbox status -u user firstsaved Junk

Amir Caspi

8 Aug 8 Aug

9:34 p.m.

Hi all,

Might anyone have any idea about this issue? I can run a cron job if needed but it seems like autoexpunge SHOULD be doing this automatically...

Thanks!

--- Amir

...

On Jul 24, 2019, at 10:18 PM, Amir Caspi Cepheid@3phase.com wrote:

Hi all,

I set up dovecot a couple of months ago and am having trouble getting autoexpunge=30d to work on my Trash and Junk mailboxes. Not sure why not because I'm not getting error messages in my log. Running "doveadm search -u <user> mailbox Junk savedbefore 30d" shows me many messages (I've got messages back to mid-May, and a couple of other users have them back to early April, although if this setting were working, there should be nothing earlier than June 24). Running a manual doveadm expunge works fine... it's just autoexpunge that seems to not be running at all.

I'm using sendmail as the MTA and procmail as the LDA, so dovecot is running purely for IMAP/POP service.

Any help is much appreciated.

Thanks!

doveconf -n: # 2.2.36 (1f10bfa63): /etc/dovecot/dovecot.conf # OS: Linux 3.10.0-957.21.3.el7.x86_64 x86_64 CentOS Linux release 7.6.1810 (Core)
# Hostname: REDACTED auth_username_format = %Ln first_valid_uid = 1000 mail_location = mbox:~/mail:INBOX=/var/spool/mail/%u mailbox_list_index = yes mbox_write_locks = fcntl namespace compat1 { alias_for = hidden = yes list = no location = prefix = mail/ separator = / } namespace compat2 { alias_for = hidden = yes list = no location = prefix = ~/mail/ separator = / } namespace compat3 { alias_for = hidden = yes list = no location = prefix = ~%u/mail/ separator = / } namespace inbox { inbox = yes location = mailbox Archive { special_use = \Archive } mailbox "Deleted Messages" { autoexpunge = 30 days special_use = \Trash } mailbox Drafts { special_use = \Drafts } mailbox Junk { autoexpunge = 30 days special_use = \Junk } mailbox "Junk E-mail" { autoexpunge = 30 days special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Spam { autoexpunge = 30 days special_use = \Junk } mailbox Trash { autoexpunge = 30 days special_use = \Trash } prefix = separator = / } passdb { driver = pam } pop3_uidl_format = %08Xv%08Xu ssl_cert = # REDACTED ssl_cipher_list = # REDACTED ssl_dh_parameters_length = # REDACTED ssl_key = # hidden, use -P to show it ssl_prefer_server_ciphers = yes userdb { driver = passwd }

--- Amir

Reio Remma

9:52 p.m.

Hello!

I have the autoexpunge settings defined inside protocol imap thus (and it works):

protocol imap { mail_plugins = quota notify replication imap_quota imap_sieve namespace inbox { location = mailbox Ham { autoexpunge = 365 days } mailbox Spam { autoexpunge = 365 days } mailbox Trash { autoexpunge = 180 days } prefix = } }

Good luck, Reio

On 08.08.2019 21:34, Amir Caspi via dovecot wrote:

...

Hi all,

Might anyone have any idea about this issue? I can run a cron job if needed but it seems like autoexpunge SHOULD be doing this automatically...

Thanks!

--- Amir

...
On Jul 24, 2019, at 10:18 PM, Amir Caspi mailto:Cepheid@3phase.com> wrote:

Hi all,

I set up dovecot a couple of months ago and am having trouble getting autoexpunge=30d to work on my Trash and Junk mailboxes. Not sure why not because I'm not getting error messages in my log. Running "doveadm search -u <user> mailbox Junk savedbefore 30d" shows me many messages (I've got messages back to mid-May, and a couple of other users have them back to early April, although if this setting were working, there should be nothing earlier than June 24). Running a manual doveadm expunge works fine... it's just autoexpunge that seems to not be running at all.

I'm using sendmail as the MTA and procmail as the LDA, so dovecot is running purely for IMAP/POP service.

Any help is much appreciated.

Thanks!

doveconf -n: # 2.2.36 (1f10bfa63): /etc/dovecot/dovecot.conf # OS: Linux 3.10.0-957.21.3.el7.x86_64 x86_64 CentOS Linux release 7.6.1810 (Core) # Hostname: REDACTED auth_username_format = %Ln first_valid_uid = 1000 mail_location = mbox:~/mail:INBOX=/var/spool/mail/%u mailbox_list_index = yes mbox_write_locks = fcntl namespace compat1 { alias_for = hidden = yes list = no location = prefix = mail/ separator = / } namespace compat2 { alias_for = hidden = yes list = no location = prefix = ~/mail/ separator = / } namespace compat3 { alias_for = hidden = yes list = no location = prefix = ~%u/mail/ separator = / } namespace inbox { inbox = yes location = mailbox Archive { special_use = \Archive } mailbox "Deleted Messages" { autoexpunge = 30 days special_use = \Trash } mailbox Drafts { special_use = \Drafts } mailbox Junk { autoexpunge = 30 days special_use = \Junk } mailbox "Junk E-mail" { autoexpunge = 30 days special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Spam { autoexpunge = 30 days special_use = \Junk } mailbox Trash { autoexpunge = 30 days special_use = \Trash } prefix = separator = / } passdb { driver = pam } pop3_uidl_format = %08Xv%08Xu ssl_cert = # REDACTED ssl_cipher_list = # REDACTED ssl_dh_parameters_length = # REDACTED ssl_key = # hidden, use -P to show it ssl_prefer_server_ciphers = yes userdb { driver = passwd }

Amir Caspi

10:03 p.m.

On Aug 8, 2019, at 12:52 PM, Reio Remma via dovecot dovecot@dovecot.org wrote:

...

I have the autoexpunge settings defined inside protocol imap thus (and it works):

It looks like my definitions are outside the protocol, as this is the default in /etc/dovecot/conf.d/15-mailboxes.conf...

MUST autoexpunge be declared within the imap protocol, even if the mailboxes themselves are declared outside of the protocol? Should the mailboxes be declared only within the protocol?

Can I declare autoexpunge for the mailboxes within the imap protocol in 20-imap.conf while leaving the main mailbox declarations in 15-mailboxes.conf? That is, will the settings "stack" or can mailbox definitions occur only in one place?

If autoexpunge can only be declared within the imap protocol -- folks who access email only through POP (and who never see Junk/Trash) will not have things auto-expunged. The Junk mailbox, in particular, is auto-populated by spamassassin via procmail so it will have messages even if the user never sees them... so it needs to be autoexpunged even if the user never logs in via imap.

IMHO the setting should apply regardless of protocol, but is that actually the case in practice?

Thanks!

--- Amir

unknown＠example.com

10 Aug 10 Aug

3:15 a.m.

On 8 Aug 2019, at 13:03, Amir Caspi cepheid@3phase.com wrote:

...

IMHO the setting should apply regardless of protocol, but is that actually the case in practice?

It seems to be broken.

I have

namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { autoexpunge = 14 days auto = subscribe special_use = \Junk } … }

and I have messages in my Junk Mail from 16 July 2019

-- I AM ZOMBOR! (kelly) ZOMBOR!

Sami Ketola

12 Aug 12 Aug

9:18 a.m.

...

On 10 Aug 2019, at 3.15, @lbutlr via dovecot dovecot@dovecot.org wrote:

On 8 Aug 2019, at 13:03, Amir Caspi cepheid@3phase.com wrote:

...
IMHO the setting should apply regardless of protocol, but is that actually the case in practice?

It seems to be broken.

I have

namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { autoexpunge = 14 days auto = subscribe special_use = \Junk } … }

and I have messages in my Junk Mail from 16 July 2019

Quoting questions from Timo:

"1. What's the first mail's saved-timestamp? doveadm fetch -u user date.saved mailbox Junk 1

That timestamp should also be the same in dovecot.list.index: doveadm mailbox status -u user firstsaved Junk"

can you check that information?

Sami

＠lbutlr

6:28 p.m.

On 12 Aug 2019, at 00:18, Sami Ketola Sami.Ketola@Open-Xchange.com wrote:

...

"1. What's the first mail's saved-timestamp? doveadm fetch -u user date.saved mailbox Junk 1

That timestamp should also be the same in dovecot.list.index: doveadm mailbox status -u user firstsaved Junk"

can you check that information?

Not for awhile since I manually cleaned out my junk a few days ago.

However, messages are moved to Junk almost entirely automatically. One or two outliers, sure, but a bunch of messages over 2 weeks old that have been in junk less than two weeks seems… unlikely.

Amir Caspi

11:01 p.m.

On Aug 12, 2019, at 12:18 AM, Sami Ketola via dovecot dovecot@dovecot.org wrote:

...

"1. What's the first mail's saved-timestamp? doveadm fetch -u user date.saved mailbox Junk 1

Well... this is giving me strange results. For myself, when I run this command, I get:

[~]# doveadm fetch -u cepheid date.saved mailbox Junk 1 date.saved: 2019-07-15 01:42:56

However, my actual Junk folder shows email from as far back as May 15. I checked this with webmail to ensure it wasn't some problem on my local MUA... this is definitely on the server. And yes, I checked the Received headers to make sure the dates were correctly represented (I didn't trust the Date header alone).

HOWEVER... it looks like something is either corrupted or otherwise messed up, because messages #2 and 299 have identical timestamps: [~]# doveadm fetch -u cepheid date.saved mailbox Junk 2 date.saved: 2019-07-25 03:54:14 [~]# doveadm fetch -u cepheid date.saved mailbox Junk 299 date.saved: 2019-07-25 03:54:14

In fact, MANY messages have this same timestamp.

I can't find the "true" first message timestamp, but the first one not in July is 107: [~]# doveadm fetch -u cepheid date.saved mailbox Junk 107 date.saved: 2019-05-29 05:57:55

This seems... not right.

The results are similarly strange for my other users.

...

That timestamp should also be the same in dovecot.list.index: doveadm mailbox status -u user firstsaved Junk"

For this I get:

[~]# doveadm mailbox status -u cepheid firstsaved Junk Junk firstsaved=1563154976

I can't tell how that timestamp corresponds to a human-readable date, however.

Cheers and thanks.

--- Amir

Thomas Zajic

13 Aug 13 Aug

5:54 a.m.

Amir Caspi via dovecot, 12.08.19 22:01

...

[~]# doveadm mailbox status -u cepheid firstsaved Junk Junk firstsaved=1563154976

I can't tell how that timestamp corresponds to a human-readable date, however.

[zlatko@disclosure:~]$ date -d @1563154976 Mon Jul 15 03:42:56 CEST 2019

HTH, Thomas

Amir Caspi

5:57 a.m.

On Aug 12, 2019, at 8:54 PM, Thomas Zajic via dovecot dovecot@dovecot.org wrote:

...

Amir Caspi via dovecot, 12.08.19 22:01

...
[~]# doveadm mailbox status -u cepheid firstsaved Junk Junk firstsaved=1563154976

I can't tell how that timestamp corresponds to a human-readable date, however.

[zlatko@disclosure:~]$ date -d @1563154976 Mon Jul 15 03:42:56 CEST 2019

So this is the same timestamp as date.saved on message 1... as it should be, I guess. Except that, as I showed, the timestamps are definitely messed up somehow. The timestamps in my MUA (whether webmail or local mail app) show just fine... so something seems to be corrupted with the timestamps in the dovecot index file, I think. But the weird thing is that this is affecting all users, not just me.

Hopefully we can figure this out.

--- Amir

Timo Sirainen

14 Aug 14 Aug

10:26 p.m.

On 13 Aug 2019, at 5.57, Amir Caspi via dovecot dovecot@dovecot.org wrote:

...

On Aug 12, 2019, at 8:54 PM, Thomas Zajic via dovecot dovecot@dovecot.org wrote:

...

Amir Caspi via dovecot, 12.08.19 22:01

...
[~]# doveadm mailbox status -u cepheid firstsaved Junk Junk firstsaved=1563154976

I can't tell how that timestamp corresponds to a human-readable date, however.

[zlatko@disclosure:~]$ date -d @1563154976 Mon Jul 15 03:42:56 CEST 2019

So this is the same timestamp as date.saved on message 1... as it should be, I guess. Except that, as I showed, the timestamps are definitely messed up somehow. The timestamps in my MUA (whether webmail or local mail app) show just fine... so something seems to be corrupted with the timestamps in the dovecot index file, I think. But the weird thing is that this is affecting all users, not just me.

It probably has something to do with using mbox format. Are the IMAP UIDs changing unexpectedly? Errors/warnings logged related to it? Unfortunately it's a rather troublesome mailbox format. There are likely some bugs in Dovecot mbox code, but it's difficult and time consuming to try to reproduce any of the bugs so I've mostly given up trying.

Amir Caspi

10:35 p.m.

On Aug 14, 2019, at 1:26 PM, Timo Sirainen via dovecot dovecot@dovecot.org wrote:

...

It probably has something to do with using mbox format. Are the IMAP UIDs changing unexpectedly? Errors/warnings logged related to it? Unfortunately it's a rather troublesome mailbox format. There are likely some bugs in Dovecot mbox code, but it's difficult and time consuming to try to reproduce any of the bugs so I've mostly given up trying.

I'm not getting any errors or warnings as far as I can tell, and I don't think the UIDs are changing unexpectedly -- messages are not getting re-downloaded randomly. That is, everything SEEMS to be working fine, as far as I can tell.

So many people still use mbox that I hope we can fix this issue.

I'm happy to help test or provide further debug output... this problem is certainly reproducible here, and it seems like lbutlr has a similar problem, so hopefully we can address at least this one...

(I'm also happy to give you the Junk mailbox and index files... there's nothing sensitive in my spam!)

Cheers.

--- Amir

Timo Sirainen

10:45 p.m.

On 14 Aug 2019, at 22.35, Amir Caspi via dovecot dovecot@dovecot.org wrote:

...

On Aug 14, 2019, at 1:26 PM, Timo Sirainen via dovecot mailto:dovecot@dovecot.org> wrote:

...
It probably has something to do with using mbox format. Are the IMAP UIDs changing unexpectedly? Errors/warnings logged related to it? Unfortunately it's a rather troublesome mailbox format. There are likely some bugs in Dovecot mbox code, but it's difficult and time consuming to try to reproduce any of the bugs so I've mostly given up trying.

I'm not getting any errors or warnings as far as I can tell, and I don't think the UIDs are changing unexpectedly -- messages are not getting re-downloaded randomly. That is, everything SEEMS to be working fine, as far as I can tell.

So many people still use mbox that I hope we can fix this issue.

I'm happy to help test or provide further debug output... this problem is certainly reproducible here, and it seems like lbutlr has a similar problem, so hopefully we can address at least this one...

(I'm also happy to give you the Junk mailbox and index files... there's nothing sensitive in my spam!)

It's not very helpful to look at the indexes after the problem already happened. But if you can find a reliably reproducible way to make this happen starting from an empty mailbox, I could look into it further. Ideally it would be a standalone script that reproduces the problem every time. Possibly something like:

Deliver mails with procmail
Read the mails with doveadm fetch
Maybe expunge the mails with doveadm expunge
Keep checking the uid and date.saved with doveadm fetch to see if they unexpectedly change at some point

Amir Caspi

11:27 p.m.

On Aug 14, 2019, at 1:45 PM, Timo Sirainen timo@sirainen.com wrote:

...

It's not very helpful to look at the indexes after the problem already happened.

Good point! I guess I'll empty out the mailboxes completely, touch a completely empty file, and see if the problem recurs. I'll shorten the autoexpunge time to 7 days so we don't have to wait a month to see if it recurs.

Thanks.

--- Amir

1922

Age (days ago)

1942

Last active (days ago)

List overview

14 comments

7 participants

participants (7)

unknown＠example.com
＠lbutlr
Amir Caspi
Reio Remma
Sami Ketola
Thomas Zajic
Timo Sirainen