[Dovecot] dovecot expire doesn't work (?)
Hi!
I'm using dovecot-1.1.5 and trying to make the expire plugin work. What I've configured in dovecot.conf is the following:
protocol imap,pop3,lda { mail_plugins = [...] expire }
dict { expire = db:/var/dovecot/expire/expire.db }
plugin { expire = spamassassin/SPAM 2 spamassassin/HAM 2 expire_dict = proxy::expire }
I have a sieve rule, to copy certain messages to my "spamassassin/SPAM" folder. Then I want to expire those messages after 2 days (I think I've configured that under the plugin{} section in dovecot.conf). So the actual message saving is done by the dovecot's deliver, but I have the plugin loaded under the "protocol lda {}" section too. So I thought now I just have to wait 2 days, and run the expire-tool, and then it will expire the messages. Now I have three messages dated back to 10.25, but running the expire-tool outputs nothing. # dovecot --exec-mail ext /usr/local/libexec/dovecot/expire-tool --test
Nothing in the logfiles, and nothing on the console. I have the /var/dovecot/expire directory: # ls -la /var/dovecot/expire/ total 1640 drwx------ 2 root wheel 512 Oct 26 19:47:53 2008 ./ drwxr-x--- 3 root wheel 512 Oct 27 07:57:42 2008 ../ -rw------- 1 root wheel 24576 Oct 27 13:00:01 2008 __db.001 -rw------- 1 root wheel 57344 Oct 27 13:00:01 2008 __db.002 -rw------- 1 root wheel 270336 Oct 27 13:00:01 2008 __db.003 -rw------- 1 root wheel 98304 Oct 27 13:00:01 2008 __db.004 -rw------- 1 root wheel 49152 Oct 27 13:00:01 2008 __db.005 -rw------- 1 root wheel 32768 Oct 26 19:47:37 2008 expire.db -rw------- 1 root wheel 10485760 Oct 27 14:22:08 2008 log.0000000001
It contains the familiar BDB files, so I think it works, although the expire.db's modify time is yesterday, but deliver saved some messages also today to the spamassassin/SPAM folder.
What could be the problem? Am I missing something?
Thanks!
Daniel
-- LEVAI Daniel PGP key ID = 0x4AC0A4B1 Key fingerprint = D037 03B9 C12D D338 4412 2D83 1373 917A 4AC0 A4B1
Hi!
Anybody has this working? I mean, at least somebody has to use this expire plugin with BDB, no? :D I think, maybe the problem is withing the plugin { expire = } setting; to be honest, I'm a little confused with this. I have a maildir layout like this: [..some dirs..]/$USER/Maildir/
Under the above directory, there are the users' subdirectories. So the subdirectory in question (spamassassin/SPAM), is "[..somedirs..]/$USER/Maildir/.spamassassin.SPAM/"
# ls -1a [..somedirs..]/$USER/Maildir/.spamassassin.SPAM/ ./ ../ cur/ dovecot-uidlist dovecot.index dovecot.index.cache dovecot.index.log maildirfolder new/ tmp/
According to the WIKI page, I must write this to my dovecot.conf: plugin { expire = spamassassin/SPAM 1 expire_dict = proxy::expire }
That is, if I want to expire the emails under the spamassassin/SPAM directory, after 1 day. Is this correct? I'm asking this because with this setup, the expire-tool does nothing... nothing in the logs either. I've experimented with "expire = spamassassin.SPAM" and "expire=spamassassin/SPAM/*" too, but no luck. Could someone send me some working dovecot.conf, and setup, so I can diff it against mine? Or maybe just a little tossing in the right direction would be enough.
Thanks in advance!
Daniel
LÉVAI Dániel wrote:
Hi!
I'm using dovecot-1.1.5 and trying to make the expire plugin work. What I've configured in dovecot.conf is the following:
protocol imap,pop3,lda { mail_plugins = [...] expire }
dict { expire = db:/var/dovecot/expire/expire.db }
plugin { expire = spamassassin/SPAM 2 spamassassin/HAM 2 expire_dict = proxy::expire }
I have a sieve rule, to copy certain messages to my "spamassassin/SPAM" folder. Then I want to expire those messages after 2 days (I think I've configured that under the plugin{} section in dovecot.conf). So the actual message saving is done by the dovecot's deliver, but I have the plugin loaded under the "protocol lda {}" section too. So I thought now I just have to wait 2 days, and run the expire-tool, and then it will expire the messages. Now I have three messages dated back to 10.25, but running the expire-tool outputs nothing. # dovecot --exec-mail ext /usr/local/libexec/dovecot/expire-tool --test
Nothing in the logfiles, and nothing on the console. I have the /var/dovecot/expire directory: # ls -la /var/dovecot/expire/ total 1640 drwx------ 2 root wheel 512 Oct 26 19:47:53 2008 ./ drwxr-x--- 3 root wheel 512 Oct 27 07:57:42 2008 ../ -rw------- 1 root wheel 24576 Oct 27 13:00:01 2008 __db.001 -rw------- 1 root wheel 57344 Oct 27 13:00:01 2008 __db.002 -rw------- 1 root wheel 270336 Oct 27 13:00:01 2008 __db.003 -rw------- 1 root wheel 98304 Oct 27 13:00:01 2008 __db.004 -rw------- 1 root wheel 49152 Oct 27 13:00:01 2008 __db.005 -rw------- 1 root wheel 32768 Oct 26 19:47:37 2008 expire.db -rw------- 1 root wheel 10485760 Oct 27 14:22:08 2008 log.0000000001
It contains the familiar BDB files, so I think it works, although the expire.db's modify time is yesterday, but deliver saved some messages also today to the spamassassin/SPAM folder.
What could be the problem? Am I missing something?
Thanks!
Daniel
-- LEVAI Daniel PGP key ID = 0x4AC0A4B1 Key fingerprint = D037 03B9 C12D D338 4412 2D83 1373 917A 4AC0 A4B1
Anybody has this working?
Yes.
plugin { expire = spamassassin/SPAM 1 expire_dict = proxy::expire }
For me it works by using '.' as hierarchy separator. So in your case expire = spamassassin.SPAM 1 should work. This could be different when using namespaces, which I don't.
That is, if I want to expire the emails under the spamassassin/SPAM directory, after 1 day. Is this correct? I'm asking this because with this setup, the expire-tool does nothing... nothing in the logs either.
Try and set mail_debug=yes. It should log more.
e-frog wrote:
Anybody has this working?
Yes.
Great! Then all hope is not lost :)
plugin { expire = spamassassin/SPAM 1 expire_dict = proxy::expire }
For me it works by using '.' as hierarchy separator. So in your case expire = spamassassin.SPAM 1 should work. This could be different when using namespaces, which I don't.
I don't use namespaces either, and I've changed my dovecot.conf to use the '.' as separator.
That is, if I want to expire the emails under the spamassassin/SPAM directory, after 1 day. Is this correct? I'm asking this because with this setup, the expire-tool does nothing... nothing in the logs either.
Try and set mail_debug=yes. It should log more.
Done.
When I ran `dovecot --exec-mail ext /usr/local/libexec/dovecot/expire-tool --test', it told me that: Info: leva/spamassassin.SPAM: stop, expire time in future: 1225290174
Since I've run it without the --test option, it outputs nothing anymore, and nothing in the logs.
What else did you configure? Or is there some other thing I should do?
Thanks!
Daniel
-- LEVAI Daniel PGP key ID = 0x4AC0A4B1 Key fingerprint = D037 03B9 C12D D338 4412 2D83 1373 917A 4AC0 A4B1
On Wed, 2008-10-29 at 15:25 +0100, LÉVAI Dániel wrote:
That is, if I want to expire the emails under the spamassassin/SPAM directory, after 1 day. Is this correct? I'm asking this because with this setup, the expire-tool does nothing... nothing in the logs either.
Try and set mail_debug=yes. It should log more. Done.
When I ran `dovecot --exec-mail ext /usr/local/libexec/dovecot/expire-tool --test', it told me that: Info: leva/spamassassin.SPAM: stop, expire time in future: 1225290174
Sounds like it's working. It just wasn't time yet to expunge the oldest mail from there:
% unix2date.sh -u 1225290174 Wed Oct 29 14:22:54 UTC 2008
Running it again now would show that it's going to expunge it? I suppose I should change the code to show the timestamp in user-understandable form. :)
On Wednesday 29 October 2008 15.53.24 you wrote:
On Wed, 2008-10-29 at 15:25 +0100, LÉVAI Dániel wrote:
That is, if I want to expire the emails under the spamassassin/SPAM directory, after 1 day. Is this correct? I'm asking this because with this setup, the expire-tool does nothing... nothing in the logs either.
Try and set mail_debug=yes. It should log more.
Done.
When I ran `dovecot --exec-mail ext /usr/local/libexec/dovecot/expire-tool --test', it told me that: Info: leva/spamassassin.SPAM: stop, expire time in future: 1225290174
Sounds like it's working. It just wasn't time yet to expunge the oldest mail from there:
% unix2date.sh -u 1225290174 Wed Oct 29 14:22:54 UTC 2008
Running it again now would show that it's going to expunge it? Unfortunately not: # ls -lrt $USER/Maildir/.spamassassin.SPAM/cur/ 1840 Oct 25 03:17:55 2008 1225137329.[...],W=1890,S=1840:2,S 1978 Oct 25 04:32:37 2008 1225137329.[...],W=2028,S=1978:2,S 3773 Oct 25 13:12:23 2008 1225137329.[...],W=3841,S=3773:2,S 2749 Oct 26 02:10:28 2008 1225137329.[...],W=2835,S=2749:2,S 3783 Oct 26 13:15:36 2008 1225137329.[...],W=3852,S=3783:2,S 13172 Oct 26 18:30:16 2008 1225137329.[...],W=13377,S=13172:2,S 2292 Oct 27 08:00:07 2008 1225137329.[...],W=2340,S=2292:2,S 1985 Oct 27 08:00:10 2008 1225137329.[...],W=2031,S=1985:2,S 26644 Oct 27 16:02:35 2008 1225137329.[...],W=27234,S=26644:2,S 3209 Oct 27 16:23:57 2008 1225137329.[...],W=3282,S=3209:2,S 3155 Oct 27 19:13:38 2008 1225290969.[...],W=3210,S=3155:2,S 3193 Oct 27 22:11:38 2008 1225291087.[...],W=3267,S=3193:2,S 4444 Oct 28 01:06:51 2008 1225203774.[...],W=4502,S=4444:2,S 4793 Oct 28 09:42:31 2008 1225203785.[...],W=4849,S=4793:2,S 2925 Oct 28 14:04:10 2008 1225203793.[...],W=2977,S=2925:2,S 3727 Oct 28 17:51:33 2008 1225218230.[...],W=3819,S=3727:2,S 255537 Oct 29 00:27:32 2008 1225236452.[...],S=255537,W=259071:2,S 4021 Oct 29 04:02:59 2008 1225249379.[...],S=4021,W=4119:2,S 2802 Oct 29 16:03:29 2008 1225292609.[...],S=2802,W=2874:2,S
You see the oldest message is dated back four days from now, and no matter if I run the expire-tool without the --test option, it won't expunge them. Although all of the messages was saved by dovecot's deliver, the expire-tool doesn't seem to recognize but one of the messages. That is the last one, created on Oct 29, 16:03.
# /usr/local/sbin/dovecot --exec-mail
ext /usr/local/libexec/dovecot/expire-tool --test
Info: leva/spamassassin.SPAM: stop, expire time in future: 1225379009
# date -r 1225379009
Thu Oct 30 16:03:29 CET 2008
Daniel
-- LEVAI Daniel PGP key ID = 0x4AC0A4B1 Key fingerprint = D037 03B9 C12D D338 4412 2D83 1373 917A 4AC0 A4B1
# /usr/local/sbin/dovecot --exec-mail
ext /usr/local/libexec/dovecot/expire-tool --test Info: leva/spamassassin.SPAM: stop, expire time in future: 1225379009 # date -r 1225379009 Thu Oct 30 16:03:29 CET 2008
Try it again tomorrow past 16:03:29 and it will expunge all your messages older than Oct 29 16:03:29. The expire database was most likely not updated till you fixed the configuration today. This means it doesn't know about the old messages.
On Wed, 2008-10-29 at 20:14 +0100, LÉVAI Dániel wrote:
Running it again now would show that it's going to expunge it? Unfortunately not: # ls -lrt $USER/Maildir/.spamassassin.SPAM/cur/ 1840 Oct 25 03:17:55 2008 1225137329.[...],W=1890,S=1840:2,S 1978 Oct 25 04:32:37 2008 1225137329.[...],W=2028,S=1978:2,S .. You see the oldest message is dated back four days from now, and no matter if I run the expire-tool without the --test option, it won't expunge them.
Ah, but expunge plugin works on "save dates", not by mtime. The save date is cached to dovecot.index.cache file and it's always older or equal to file's ctime. So use ls -lc instead.
On Monday 27 October 2008 14.26.50 LÉVAI Dániel wrote:
Hi!
I'm using dovecot-1.1.5 and trying to make the expire plugin work. What I've configured in dovecot.conf is the following:
protocol imap,pop3,lda { mail_plugins = [...] expire }
dict { expire = db:/var/dovecot/expire/expire.db }
plugin { expire = spamassassin/SPAM 2 spamassassin/HAM 2 expire_dict = proxy::expire }
I have a sieve rule, to copy certain messages to my "spamassassin/SPAM" folder. Then I want to expire those messages after 2 days (I think I've configured that under the plugin{} section in dovecot.conf). So the actual message saving is done by the dovecot's deliver, but I have the plugin loaded under the "protocol lda {}" section too. So I thought now I just have to wait 2 days, and run the expire-tool, and then it will expire the messages. Now I have three messages dated back to 10.25, but running the expire-tool outputs nothing. # dovecot --exec-mail ext /usr/local/libexec/dovecot/expire-tool --test
Nothing in the logfiles, and nothing on the console. I have the /var/dovecot/expire directory: # ls -la /var/dovecot/expire/ total 1640 drwx------ 2 root wheel 512 Oct 26 19:47:53 2008 ./ drwxr-x--- 3 root wheel 512 Oct 27 07:57:42 2008 ../ -rw------- 1 root wheel 24576 Oct 27 13:00:01 2008 __db.001 -rw------- 1 root wheel 57344 Oct 27 13:00:01 2008 __db.002 -rw------- 1 root wheel 270336 Oct 27 13:00:01 2008 __db.003 -rw------- 1 root wheel 98304 Oct 27 13:00:01 2008 __db.004 -rw------- 1 root wheel 49152 Oct 27 13:00:01 2008 __db.005 -rw------- 1 root wheel 32768 Oct 26 19:47:37 2008 expire.db -rw------- 1 root wheel 10485760 Oct 27 14:22:08 2008 log.0000000001
It contains the familiar BDB files, so I think it works, although the expire.db's modify time is yesterday, but deliver saved some messages also today to the spamassassin/SPAM folder.
I've got bitten by this: The wiki[1] reads:
[...] plugin {
- "%" works by matching any number of characters, but it stops at the hierarchy separator. Currently the separator is hardcoded to "/". [...]
# Trash and its children 7d, Spam 30d expire = Trash 7 Trash/* 7 Spam 30 [...]
That is not exactly true. The separator which is working (as told me by e-frog, and as can be seen in the Maildir/ hierarchy) is the dot character (ie.: .). My $USER/spamassassin/SPAM directory is not working as: expire = spamassassin/SPAM 1 only as: expire = spamassassin.SPAM 1
Also the dovecot-example.conf says: "The following dict block maps dictionary names to URIs when the server is used. These can then be referenced using URIs in format "proxy:<name>"."
That is not true either, it must be "proxy::<name>" (note the two colons) or else dovecot won't even start.
Anyway, for the record, I should mention that while it is easy to check whether dovecot is fooling around with a mysql database, it is not so straightforward with BDB. One can check if the Berkeley database is being used with db4_dump (or on some systems db4.7_dump or db4.6_dump and so on...):
$ db4_dump -d a expire.db [...] page 1: btree leaf: LSN [0][1]: level 1 prev: 0 next: 0 entries: 0 offset: 16384
^^^ the above contains no entries, while: $ db4_dump -da expire.db [...] page 1: btree leaf: LSN [1][84670]: level 1 prev: 0 next: 0 entries: 2 offset: 16344 [000] 16352 len: 29 data: shared/leva/spamassa... [001] 16344 len: 4 data: ��0x08I ^^^ this contains entries. Don't ask me what is the second row, though :), and also it is a PITA that the data gets trimmed.
On Wednesday 29 October 2008 15.53.24 Timo Sirainen wrote:
On Wed, 2008-10-29 at 15:25 +0100, LÉVAI Dániel wrote:
When I ran `dovecot --exec-mail ext /usr/local/libexec/dovecot/expire-tool --test', it told me that: Info: leva/spamassassin.SPAM: stop, expire time in future: 1225290174
Sounds like it's working. It just wasn't time yet to expunge the oldest mail from there
Yep, now I can understand that, but what *is* weird, that the only "debug" information comes from this expire-tool when ran with the --test option. If I run it without it, it won't output anything anywhere. It would be nice to increase the logging for this (with or without the --test option), eg. when mail_debug=yes.
[1] - http://wiki.dovecot.org/Plugins/Expire
Daniel
-- LEVAI Daniel PGP key ID = 0x4AC0A4B1 Key fingerprint = D037 03B9 C12D D338 4412 2D83 1373 917A 4AC0 A4B1
On Thu, 2008-10-30 at 16:28 +0100, LÉVAI Dániel wrote:
I've got bitten by this: The wiki[1] reads:
[...] plugin {
- "%" works by matching any number of characters, but it stops at the hierarchy separator. Currently the separator is hardcoded to "/". [...]
# Trash and its children 7d, Spam 30d expire = Trash 7 Trash/* 7 Spam 30 [...]
That is not exactly true. The separator which is working (as told me by e-frog, and as can be seen in the Maildir/ hierarchy) is the dot character (ie.: .).
The "/" hardcoding only means the "%" wildcard matching, meaning if you've a mailbox "foo/bar" then "%" would match only "foo" part, but if you've a mailbox "foo.bar" then "%" would match the full "foo.bar". In any case you'll need to use the separator you've configured in your namespaces. I guess wiki should explain this clearly. Or better yet, I could fix the whole issue and remove it from wiki. :)
Also the dovecot-example.conf says: "The following dict block maps dictionary names to URIs when the server is used. These can then be referenced using URIs in format "proxy:<name>"."
That is not true either, it must be "proxy::<name>" (note the two colons) or else dovecot won't even start.
Fixed now.
Yep, now I can understand that, but what *is* weird, that the only "debug" information comes from this expire-tool when ran with the --test option. If I run it without it, it won't output anything anywhere. It would be nice to increase the logging for this (with or without the --test option), eg. when mail_debug=yes.
What should it write? I guess -v parameter could do something. mail_debug=yes could affect the plugin's logging.
On Thursday 30 October 2008 16.42.16 Timo Sirainen wrote:
On Thu, 2008-10-30 at 16:28 +0100, LÉVAI Dániel wrote:
I've got bitten by this: The wiki[1] reads:
[...] the hierarchy separator. Currently the separator is hardcoded to
- "%" works by matching any number of characters, but it stops at
"/". [...] plugin { # Trash and its children 7d, Spam 30d expire = Trash 7 Trash/* 7 Spam 30 [...]
That is not exactly true. The separator which is working (as told me by e-frog, and as can be seen in the Maildir/ hierarchy) is the dot character (ie.: .).
The "/" hardcoding only means the "%" wildcard matching, meaning if you've a mailbox "foo/bar" then "%" would match only "foo" part, but if you've a mailbox "foo.bar" then "%" would match the full "foo.bar". In any case you'll need to use the separator you've configured in your namespaces. Where have I configured that? I'm just using maildir: in the mail_location, and if I create a subdirectory with my MUA, in the server on the filesystem it will separate it from the parent with a dot. Is this configurable?
Yep, now I can understand that, but what *is* weird, that the only "debug" information comes from this expire-tool when ran with the --test option. If I run it without it, it won't output anything anywhere. It would be nice to increase the logging for this (with or without the --test option), eg. when mail_debug=yes.
What should it write? I guess -v parameter could do something. mail_debug=yes could affect the plugin's logging. I think, it should display that it found an "expire = something" entry in dovecot.conf, and that it could find a matching directory under the mail_location. While iterating over the messages that it has found, it would be nice if it would write an info line with each message, including the message's path/name, the message's expire date in the future, that it has been expunged, or that it would have been expunged, but the --test option was set. Like: $ .../expire-tool --test expire = spamassassin.SPAM 1 spamassassin.HAM 1 match: /var/virtualmaildir/user1/Maildir/.spamassassin.SPAM/ cur/1225290969.M266240P9922.host,W=3210,S=3155:2,S will expire on Oct 31, 13:01 cur/1225291087.M157951P9922.host,W=3267,S=3193:2,S will expire on Oct 30, 19:25 cur/1225292609.M646577P6712.host,S=2802,W=2874:2,S expunged (not really) cur/1225316456.M35928P11573.host,S=3760,W=3852:2,S expunged (not really) cur/1225333013.M644866P4387.host,S=4208,W=4311:2,S expunged (not really) cur/1225350074.M658507P24283.host,W=2508,S=2462:2,S expunged (not really) cur/1225361450.M896405P30952.host,S=58036,W=58865:2,S expunged (not really) match: /var/virtualmaildir/user1/Maildir/.spamassassin.HAM/ cur/1225381029.M654813P11449.host,W=5325,S=5268:2,S expunged (not really) match: /var/virtualmaildir/user116/Maildir/.spamassassin.HAM/ cur/1225290969.M35928P11573.host,W=5325,S=5268:2,S will expire on Oct 31, 01:54
Of course without the --test option it wouldn't have to write the (not really) part. Anyway, something like the above :)
Daniel
-- LEVAI Daniel PGP key ID = 0x4AC0A4B1 Key fingerprint = D037 03B9 C12D D338 4412 2D83 1373 917A 4AC0 A4B1
On Thu, 2008-10-30 at 17:50 +0100, LÉVAI Dániel wrote:
The "/" hardcoding only means the "%" wildcard matching, meaning if you've a mailbox "foo/bar" then "%" would match only "foo" part, but if you've a mailbox "foo.bar" then "%" would match the full "foo.bar". In any case you'll need to use the separator you've configured in your namespaces. Where have I configured that? I'm just using maildir: in the mail_location,
If you haven't configured any namespaces, Dovecot will use the default namespace with the default maildir separator (dot).
and if I create a subdirectory with my MUA, in the server on the filesystem it will separate it from the parent with a dot. Is this configurable?
No. The filesystem separator is always a dot. But you can configure the "virtual separator" with namespaces, i.e. what the clients will see and what you'll use in configuration files to specify the mailbox names.
Yep, now I can understand that, but what *is* weird, that the only "debug" information comes from this expire-tool when ran with the --test option. If I run it without it, it won't output anything anywhere. It would be nice to increase the logging for this (with or without the --test option), eg. when mail_debug=yes.
What should it write? I guess -v parameter could do something. mail_debug=yes could affect the plugin's logging. I think, it should display that it found an "expire = something" entry in dovecot.conf, and that it could find a matching directory under the mail_location. While iterating over the messages that it has found, it would be nice if it would write an info line with each message, including the message's path/name,
Printing the message's path works only with maildir. Expire plugin works also with mbox. It could print message's UID number though.
the message's expire date in the future, that it has been expunged, or that it would have been expunged, but the --test option was set.
I'll see about adding these.
participants (3)
-
e-frog
-
LÉVAI Dániel
-
Timo Sirainen