Autoexpunge not working for Junk?
Hi all,
I set up dovecot a couple of months ago and am having trouble getting autoexpunge=30d to work on my Trash and Junk mailboxes. Not sure why not because I'm not getting error messages in my log.
Running "doveadm search -u <user> mailbox Junk savedbefore 30d" shows me many messages (I've got messages back to mid-May, and a couple of other users have them back to early April, although if this setting were working, there should be nothing earlier than June 24). Running a manual doveadm expunge works fine... it's just autoexpunge that seems to not be running at all.I'm using sendmail as the MTA and procmail as the LDA, so dovecot is running purely for IMAP/POP service.
Any help is much appreciated.
Thanks!
doveconf -n:
# 2.2.36 (1f10bfa63): /etc/dovecot/dovecot.conf
# OS: Linux 3.10.0-957.21.3.el7.x86_64 x86_64 CentOS Linux release 7.6.1810 (Core)
# Hostname: REDACTED
auth_username_format = %Ln
first_valid_uid = 1000
mail_location = mbox:~/mail:INBOX=/var/spool/mail/%u
mailbox_list_index = yes
mbox_write_locks = fcntl
namespace compat1 {
alias_for =
hidden = yes
list = no
location =
prefix = mail/
separator = /
}
namespace compat2 {
alias_for =
hidden = yes
list = no
location =
prefix = ~/mail/
separator = /
}
namespace compat3 {
alias_for =
hidden = yes
list = no
location =
prefix = ~%u/mail/
separator = /
}
namespace inbox {
inbox = yes
location =
mailbox Archive {
special_use = \Archive
}
mailbox "Deleted Messages" {
autoexpunge = 30 days
special_use = \Trash
}
mailbox Drafts {
special_use = \Drafts
}
mailbox Junk {
autoexpunge = 30 days
special_use = \Junk
}
mailbox "Junk E-mail" {
autoexpunge = 30 days
special_use = \Junk
}
mailbox Sent {
special_use = \Sent
}
mailbox "Sent Messages" {
special_use = \Sent
}
mailbox Spam {
autoexpunge = 30 days
special_use = \Junk
}
mailbox Trash {
autoexpunge = 30 days
special_use = \Trash
}
prefix =
separator = /
}
passdb {
driver = pam
}
pop3_uidl_format = %08Xv%08Xu
ssl_cert = # REDACTED
ssl_cipher_list = # REDACTED
ssl_dh_parameters_length = # REDACTED
ssl_key = # hidden, use -P to show it
ssl_prefer_server_ciphers = yes
userdb {
driver = passwd
}
--- Amir
On 25 Jul 2019, at 7.18, Amir Caspi via dovecot <dovecot@dovecot.org> wrote:
Hi all,
I set up dovecot a couple of months ago and am having trouble getting autoexpunge=30d to work on my Trash and Junk mailboxes. Not sure why not because I'm not getting error messages in my log. Running "doveadm search -u <user> mailbox Junk savedbefore 30d" shows me many messages (I've got messages back to mid-May, and a couple of other users have them back to early April, although if this setting were working, there should be nothing earlier than June 24). Running a manual doveadm expunge works fine... it's just autoexpunge that seems to not be running at all.
Autoexpunging tries to be efficient, so it looks only at the first email's saved-timestamp. It's also cached in dovecot.list.index. So you should check:
What's the first mail's saved-timestamp? doveadm fetch -u user date.saved mailbox Junk 1
That timestamp should also be the same in dovecot.list.index: doveadm mailbox status -u user firstsaved Junk
Hi all,
Might anyone have any idea about this issue? I can run a cron job if needed but it seems like autoexpunge SHOULD be doing this automatically...
Thanks!
--- Amir
On Jul 24, 2019, at 10:18 PM, Amir Caspi <Cepheid@3phase.com> wrote:
Hi all,
I set up dovecot a couple of months ago and am having trouble getting autoexpunge=30d to work on my Trash and Junk mailboxes. Not sure why not because I'm not getting error messages in my log. Running "doveadm search -u <user> mailbox Junk savedbefore 30d" shows me many messages (I've got messages back to mid-May, and a couple of other users have them back to early April, although if this setting were working, there should be nothing earlier than June 24). Running a manual doveadm expunge works fine... it's just autoexpunge that seems to not be running at all.
I'm using sendmail as the MTA and procmail as the LDA, so dovecot is running purely for IMAP/POP service.
Any help is much appreciated.
Thanks!
doveconf -n: # 2.2.36 (1f10bfa63): /etc/dovecot/dovecot.conf # OS: Linux 3.10.0-957.21.3.el7.x86_64 x86_64 CentOS Linux release 7.6.1810 (Core)
# Hostname: REDACTED auth_username_format = %Ln first_valid_uid = 1000 mail_location = mbox:~/mail:INBOX=/var/spool/mail/%u mailbox_list_index = yes mbox_write_locks = fcntl namespace compat1 { alias_for = hidden = yes list = no location = prefix = mail/ separator = / } namespace compat2 { alias_for = hidden = yes list = no location = prefix = ~/mail/ separator = / } namespace compat3 { alias_for = hidden = yes list = no location = prefix = ~%u/mail/ separator = / } namespace inbox { inbox = yes location = mailbox Archive { special_use = \Archive } mailbox "Deleted Messages" { autoexpunge = 30 days special_use = \Trash } mailbox Drafts { special_use = \Drafts } mailbox Junk { autoexpunge = 30 days special_use = \Junk } mailbox "Junk E-mail" { autoexpunge = 30 days special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Spam { autoexpunge = 30 days special_use = \Junk } mailbox Trash { autoexpunge = 30 days special_use = \Trash } prefix = separator = / } passdb { driver = pam } pop3_uidl_format = %08Xv%08Xu ssl_cert = # REDACTED ssl_cipher_list = # REDACTED ssl_dh_parameters_length = # REDACTED ssl_key = # hidden, use -P to show it ssl_prefer_server_ciphers = yes userdb { driver = passwd }--- Amir
Hello!
I have the autoexpunge settings defined inside protocol imap thus (and it works):
protocol imap { mail_plugins = quota notify replication imap_quota imap_sieve namespace inbox { location = mailbox Ham { autoexpunge = 365 days } mailbox Spam { autoexpunge = 365 days } mailbox Trash { autoexpunge = 180 days } prefix = } }
Good luck, Reio
On 08.08.2019 21:34, Amir Caspi via dovecot wrote:
Hi all,
Might anyone have any idea about this issue? I can run a cron job if needed but it seems like autoexpunge SHOULD be doing this automatically...
Thanks!
--- Amir
On Jul 24, 2019, at 10:18 PM, Amir Caspi <Cepheid@3phase.com <mailto:Cepheid@3phase.com>> wrote:
Hi all,
I set up dovecot a couple of months ago and am having trouble getting autoexpunge=30d to work on my Trash and Junk mailboxes. Not sure why not because I'm not getting error messages in my log. Running "doveadm search -u <user> mailbox Junk savedbefore 30d" shows me many messages (I've got messages back to mid-May, and a couple of other users have them back to early April, although if this setting were working, there should be nothing earlier than June 24). Running a manual doveadm expunge works fine... it's just autoexpunge that seems to not be running at all.
I'm using sendmail as the MTA and procmail as the LDA, so dovecot is running purely for IMAP/POP service.
Any help is much appreciated.
Thanks!
doveconf -n: # 2.2.36 (1f10bfa63): /etc/dovecot/dovecot.conf # OS: Linux 3.10.0-957.21.3.el7.x86_64 x86_64 CentOS Linux release 7.6.1810 (Core) # Hostname: REDACTED auth_username_format = %Ln first_valid_uid = 1000 mail_location = mbox:~/mail:INBOX=/var/spool/mail/%u mailbox_list_index = yes mbox_write_locks = fcntl namespace compat1 { alias_for = hidden = yes list = no location = prefix = mail/ separator = / } namespace compat2 { alias_for = hidden = yes list = no location = prefix = ~/mail/ separator = / } namespace compat3 { alias_for = hidden = yes list = no location = prefix = ~%u/mail/ separator = / } namespace inbox { inbox = yes location = mailbox Archive { special_use = \Archive } mailbox "Deleted Messages" { autoexpunge = 30 days special_use = \Trash } mailbox Drafts { special_use = \Drafts } mailbox Junk { autoexpunge = 30 days special_use = \Junk } mailbox "Junk E-mail" { autoexpunge = 30 days special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Spam { autoexpunge = 30 days special_use = \Junk } mailbox Trash { autoexpunge = 30 days special_use = \Trash } prefix = separator = / } passdb { driver = pam } pop3_uidl_format = %08Xv%08Xu ssl_cert = # REDACTED ssl_cipher_list = # REDACTED ssl_dh_parameters_length = # REDACTED ssl_key = # hidden, use -P to show it ssl_prefer_server_ciphers = yes userdb { driver = passwd }
On Aug 8, 2019, at 12:52 PM, Reio Remma via dovecot <dovecot@dovecot.org> wrote:
I have the autoexpunge settings defined inside protocol imap thus (and it works):
It looks like my definitions are outside the protocol, as this is the default in /etc/dovecot/conf.d/15-mailboxes.conf...
MUST autoexpunge be declared within the imap protocol, even if the mailboxes themselves are declared outside of the protocol? Should the mailboxes be declared only within the protocol?
Can I declare autoexpunge for the mailboxes within the imap protocol in 20-imap.conf while leaving the main mailbox declarations in 15-mailboxes.conf? That is, will the settings "stack" or can mailbox definitions occur only in one place?
If autoexpunge can only be declared within the imap protocol -- folks who access email only through POP (and who never see Junk/Trash) will not have things auto-expunged. The Junk mailbox, in particular, is auto-populated by spamassassin via procmail so it will have messages even if the user never sees them... so it needs to be autoexpunged even if the user never logs in via imap.
IMHO the setting should apply regardless of protocol, but is that actually the case in practice?
Thanks!
--- Amir
On 8 Aug 2019, at 13:03, Amir Caspi <cepheid@3phase.com> wrote:
IMHO the setting should apply regardless of protocol, but is that actually the case in practice?
It seems to be broken.
I have
namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { autoexpunge = 14 days auto = subscribe special_use = \Junk } … }
and I have messages in my Junk Mail from 16 July 2019
-- I AM ZOMBOR! (kelly) ZOMBOR!
On 10 Aug 2019, at 3.15, @lbutlr via dovecot <dovecot@dovecot.org> wrote:
On 8 Aug 2019, at 13:03, Amir Caspi <cepheid@3phase.com> wrote:
IMHO the setting should apply regardless of protocol, but is that actually the case in practice?
It seems to be broken.
I have
namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { autoexpunge = 14 days auto = subscribe special_use = \Junk } … }
and I have messages in my Junk Mail from 16 July 2019
Quoting questions from Timo:
"1. What's the first mail's saved-timestamp? doveadm fetch -u user date.saved mailbox Junk 1
- That timestamp should also be the same in dovecot.list.index: doveadm mailbox status -u user firstsaved Junk"
can you check that information?
Sami
On 12 Aug 2019, at 00:18, Sami Ketola <Sami.Ketola@Open-Xchange.com> wrote:
"1. What's the first mail's saved-timestamp? doveadm fetch -u user date.saved mailbox Junk 1
- That timestamp should also be the same in dovecot.list.index: doveadm mailbox status -u user firstsaved Junk"
can you check that information?
Not for awhile since I manually cleaned out my junk a few days ago.
However, messages are moved to Junk almost entirely automatically. One or two outliers, sure, but a bunch of messages over 2 weeks old that have been in junk less than two weeks seems… unlikely.
On Aug 12, 2019, at 12:18 AM, Sami Ketola via dovecot <dovecot@dovecot.org> wrote:
"1. What's the first mail's saved-timestamp? doveadm fetch -u user date.saved mailbox Junk 1
Well... this is giving me strange results. For myself, when I run this command, I get:
[~]# doveadm fetch -u cepheid date.saved mailbox Junk 1 date.saved: 2019-07-15 01:42:56
However, my actual Junk folder shows email from as far back as May 15. I checked this with webmail to ensure it wasn't some problem on my local MUA... this is definitely on the server. And yes, I checked the Received headers to make sure the dates were correctly represented (I didn't trust the Date header alone).
HOWEVER... it looks like something is either corrupted or otherwise messed up, because messages #2 and 299 have identical timestamps: [~]# doveadm fetch -u cepheid date.saved mailbox Junk 2 date.saved: 2019-07-25 03:54:14 [~]# doveadm fetch -u cepheid date.saved mailbox Junk 299 date.saved: 2019-07-25 03:54:14
In fact, MANY messages have this same timestamp.
I can't find the "true" first message timestamp, but the first one not in July is 107: [~]# doveadm fetch -u cepheid date.saved mailbox Junk 107 date.saved: 2019-05-29 05:57:55
This seems... not right.
The results are similarly strange for my other users.
- That timestamp should also be the same in dovecot.list.index: doveadm mailbox status -u user firstsaved Junk"
For this I get:
[~]# doveadm mailbox status -u cepheid firstsaved Junk Junk firstsaved=1563154976
I can't tell how that timestamp corresponds to a human-readable date, however.
Cheers and thanks.
--- Amir
- Amir Caspi via dovecot, 12.08.19 22:01
[~]# doveadm mailbox status -u cepheid firstsaved Junk Junk firstsaved=1563154976
I can't tell how that timestamp corresponds to a human-readable date, however.
[zlatko@disclosure:~]$ date -d @1563154976 Mon Jul 15 03:42:56 CEST 2019
HTH, Thomas
On Aug 12, 2019, at 8:54 PM, Thomas Zajic via dovecot <dovecot@dovecot.org> wrote:
- Amir Caspi via dovecot, 12.08.19 22:01
[~]# doveadm mailbox status -u cepheid firstsaved Junk Junk firstsaved=1563154976
I can't tell how that timestamp corresponds to a human-readable date, however.
[zlatko@disclosure:~]$ date -d @1563154976 Mon Jul 15 03:42:56 CEST 2019
So this is the same timestamp as date.saved on message 1... as it should be, I guess. Except that, as I showed, the timestamps are definitely messed up somehow. The timestamps in my MUA (whether webmail or local mail app) show just fine... so something seems to be corrupted with the timestamps in the dovecot index file, I think. But the weird thing is that this is affecting all users, not just me.
Hopefully we can figure this out.
--- Amir
On 13 Aug 2019, at 5.57, Amir Caspi via dovecot <dovecot@dovecot.org> wrote:
On Aug 12, 2019, at 8:54 PM, Thomas Zajic via dovecot <dovecot@dovecot.org> wrote:
- Amir Caspi via dovecot, 12.08.19 22:01
[~]# doveadm mailbox status -u cepheid firstsaved Junk Junk firstsaved=1563154976
I can't tell how that timestamp corresponds to a human-readable date, however.
[zlatko@disclosure:~]$ date -d @1563154976 Mon Jul 15 03:42:56 CEST 2019
So this is the same timestamp as date.saved on message 1... as it should be, I guess. Except that, as I showed, the timestamps are definitely messed up somehow. The timestamps in my MUA (whether webmail or local mail app) show just fine... so something seems to be corrupted with the timestamps in the dovecot index file, I think. But the weird thing is that this is affecting all users, not just me.
It probably has something to do with using mbox format. Are the IMAP UIDs changing unexpectedly? Errors/warnings logged related to it? Unfortunately it's a rather troublesome mailbox format. There are likely some bugs in Dovecot mbox code, but it's difficult and time consuming to try to reproduce any of the bugs so I've mostly given up trying.
On Aug 14, 2019, at 1:26 PM, Timo Sirainen via dovecot <dovecot@dovecot.org> wrote:
It probably has something to do with using mbox format. Are the IMAP UIDs changing unexpectedly? Errors/warnings logged related to it? Unfortunately it's a rather troublesome mailbox format. There are likely some bugs in Dovecot mbox code, but it's difficult and time consuming to try to reproduce any of the bugs so I've mostly given up trying.
I'm not getting any errors or warnings as far as I can tell, and I don't think the UIDs are changing unexpectedly -- messages are not getting re-downloaded randomly. That is, everything SEEMS to be working fine, as far as I can tell.
So many people still use mbox that I hope we can fix this issue.
I'm happy to help test or provide further debug output... this problem is certainly reproducible here, and it seems like lbutlr has a similar problem, so hopefully we can address at least this one...
(I'm also happy to give you the Junk mailbox and index files... there's nothing sensitive in my spam!)
Cheers.
--- Amir
On 14 Aug 2019, at 22.35, Amir Caspi via dovecot <dovecot@dovecot.org> wrote:
On Aug 14, 2019, at 1:26 PM, Timo Sirainen via dovecot <dovecot@dovecot.org <mailto:dovecot@dovecot.org>> wrote:
It probably has something to do with using mbox format. Are the IMAP UIDs changing unexpectedly? Errors/warnings logged related to it? Unfortunately it's a rather troublesome mailbox format. There are likely some bugs in Dovecot mbox code, but it's difficult and time consuming to try to reproduce any of the bugs so I've mostly given up trying.
I'm not getting any errors or warnings as far as I can tell, and I don't think the UIDs are changing unexpectedly -- messages are not getting re-downloaded randomly. That is, everything SEEMS to be working fine, as far as I can tell.
So many people still use mbox that I hope we can fix this issue.
I'm happy to help test or provide further debug output... this problem is certainly reproducible here, and it seems like lbutlr has a similar problem, so hopefully we can address at least this one...
(I'm also happy to give you the Junk mailbox and index files... there's nothing sensitive in my spam!)
It's not very helpful to look at the indexes after the problem already happened. But if you can find a reliably reproducible way to make this happen starting from an empty mailbox, I could look into it further. Ideally it would be a standalone script that reproduces the problem every time. Possibly something like:
- Deliver mails with procmail
- Read the mails with doveadm fetch
- Maybe expunge the mails with doveadm expunge
- Keep checking the uid and date.saved with doveadm fetch to see if they unexpectedly change at some point
On Aug 14, 2019, at 1:45 PM, Timo Sirainen <timo@sirainen.com> wrote:
It's not very helpful to look at the indexes after the problem already happened.
Good point! I guess I'll empty out the mailboxes completely, touch a completely empty file, and see if the problem recurs. I'll shorten the autoexpunge time to 7 days so we don't have to wait a month to see if it recurs.
Thanks.
--- Amir
participants (7)
-
unknown＠example.com
-
＠lbutlr
-
Amir Caspi
-
Reio Remma
-
Sami Ketola
-
Thomas Zajic
-
Timo Sirainen