[Dovecot] Can expire-tool skip folders with "expire time in future" errors?
Hi folks,
when I run expire-tool, I often see messages like the following:
# dovecot --exec-mail ext /usr/libexec/dovecot/expire-tool Info: user/Trash: stop, expire time in future: Fri May 22 17:42:23 2009
the server's dovecot.conf contains the following line:
expire = Trash 2 Junk 7 */News 30 */Reports 7
From what I observe, an error of "expire time in future" in one of the
Trash folders stops expire-tool alltogether, meaning that remaining folders like News or Reports don't get processed at all?
Is this interpretation correct? If so, can I convince expire-tool to skip errors and continue processing other folders?
-R
On Wed, 2009-05-20 at 18:53 +0200, Ralph Seichter wrote:
Hi folks,
when I run expire-tool, I often see messages like the following:
# dovecot --exec-mail ext /usr/libexec/dovecot/expire-tool Info: user/Trash: stop, expire time in future: Fri May 22 17:42:23 2009
It should log that only when you use --test parameter. If it happens also without --test, what dovecot version are you using?
Anyway it's not an error. It just means that all the rest of the mailboxes have future timestamps, and expire-tool has finished its work.
Timo Sirainen wrote:
Info: user/Trash: stop, expire time in future: [...]
It should log that only when you use --test parameter.
You're right, I've used the --test parameter to see what's going on.
Anyway it's not an error. It just means that all the rest of the mailboxes have future timestamps, and expire-tool has finished its work.
I have folders containing messages which have been moved there -- either by the sieve plugin or manually in Thunderbird -- days or even weeks ago. The messages are not removed even with an expire time of "somefolder 1".
So far, I thought that the expire plugin stops when the first "expire time in future" ist detected, but if I understand you correctly, this message implicitely means that there are no more messages at all to be expired?
What can I do to find out why some folders don't seem to be processed by the expire plugin? Is there any debugging/logging data I can look into?
-R
On Tue, 2009-05-26 at 13:03 +0200, Ralph Seichter wrote:
I have folders containing messages which have been moved there -- either by the sieve plugin or manually in Thunderbird -- days or even weeks ago. The messages are not removed even with an expire time of "somefolder 1".
Were the messages moved there before expire plugin was loaded? Those aren't tracked. Also if you change the expire time, it won't change immediately because the old value was already used to set the "next expire stamp" in the database.
What can I do to find out why some folders don't seem to be processed by the expire plugin? Is there any debugging/logging data I can look into?
I think it works, but you'll just have to wait for a few weeks to get the database up to date..
On 5/31/2009, Timo Sirainen (tss@iki.fi) wrote:
Were the messages moved there before expire plugin was loaded? Those aren't tracked.
Is there a simple way to fix this? Ie, maybe a script that could be run the first time a folder is added to the list, to add all of the current messages as of that date?
On Mon, 2009-06-01 at 06:33 -0400, Charles Marcus wrote:
On 5/31/2009, Timo Sirainen (tss@iki.fi) wrote:
Were the messages moved there before expire plugin was loaded? Those aren't tracked.
Is there a simple way to fix this? Ie, maybe a script that could be run the first time a folder is added to the list, to add all of the current messages as of that date?
Does it really matter if it just takes a couple of weeks to start working? Seems like too much extra work for little gain. :)
On 6/1/2009 12:40 PM, Timo Sirainen wrote:
Were the messages moved there before expire plugin was loaded? Those aren't tracked.
Is there a simple way to fix this? Ie, maybe a script that could be run the first time a folder is added to the list, to add all of the current messages as of that date?
Does it really matter if it just takes a couple of weeks to start working? Seems like too much extra work for little gain. :)
I'm thinking about when expire is enabled on a folder that already has lots of mail in it...
But I guess this could just be scripted by the local admin when needed... but what would be the proper way to do this? Would a simple 'touch' do it without breaking anything?
On Mon, 2009-06-01 at 13:09 -0400, Charles Marcus wrote:
On 6/1/2009 12:40 PM, Timo Sirainen wrote:
Were the messages moved there before expire plugin was loaded? Those aren't tracked.
Is there a simple way to fix this? Ie, maybe a script that could be run the first time a folder is added to the list, to add all of the current messages as of that date?
Does it really matter if it just takes a couple of weeks to start working? Seems like too much extra work for little gain. :)
I'm thinking about when expire is enabled on a folder that already has lots of mail in it...
But does it matter if they're deleted immediately or only 2 weeks from now? It gets done eventually anyway.
But I guess this could just be scripted by the local admin when needed... but what would be the proper way to do this? Would a simple 'touch' do it without breaking anything?
Expire plugin won't add old mails to the database no matter what you do. Admin could of course just manually delete those old mails.
Timo Sirainen wrote:
I have folders containing messages which have been moved there -- either by the sieve plugin or manually in Thunderbird -- days or even weeks ago. The messages are not removed even with an expire time of "somefolder 1".
Were the messages moved there before expire plugin was loaded? Those aren't tracked.
No, I enabled both the sieve and expire plugin at the same time.
-R
Timo Sirainen wrote:
Anyway it's not an error. It just means that all the rest of the mailboxes have future timestamps, and expire-tool has finished its work.
I upgraded to Dovecot 1.1.15, but expire still gives me trouble. I keep seeing the following error repeatedly now:
dict: db(secondary, ): unable to allocate space from the buffer cache dict: sdb.open() failed: Cannot allocate memory dict: dict: db(/var/lib/dovecot/expire.db) open failed dict: Failed to initialize dictionary 'expire'
When I run the expire-tool manually or by cron job, it dies with a segmentation fault, so effectively I currently cannot use expire at all.
-R
On Fri, 2009-05-29 at 17:26 +0200, Ralph Seichter wrote:
Timo Sirainen wrote:
Anyway it's not an error. It just means that all the rest of the mailboxes have future timestamps, and expire-tool has finished its work.
I upgraded to Dovecot 1.1.15, but expire still gives me trouble. I keep seeing the following error repeatedly now:
dict: db(secondary, ): unable to allocate space from the buffer cache dict: sdb.open() failed: Cannot allocate memory
Does the attached patch help? If not, increase the 1024*1024 to a larger value.
Timo Sirainen wrote:
Does the attached patch help? If not, increase the 1024*1024 to a larger value.
Sorry for replying so late, the server running Dovecot succumbed to a hardware problem. Now that the machine is online again, I applied your patch. Running the expire tool now causes the following log messages:
dict: db(secondary, ): DB->set_cachesize: method not permitted when environment specified dict: db(secondary, ): unable to allocate space from the buffer cache dict: sdb.open() failed: Cannot allocate memory dict: dict: db(/var/lib/dovecot/expire.db) open failed dict: Failed to initialize dictionary 'expire'
This does not look too well yet. ;-)
-R
On Thu, 2009-06-04 at 22:02 +0200, Ralph Seichter wrote:
Sorry for replying so late, the server running Dovecot succumbed to a hardware problem. Now that the machine is online again, I applied your patch. Running the expire tool now causes the following log messages:
dict: db(secondary, ): DB->set_cachesize: method not permitted when environment specified dict: db(secondary, ): unable to allocate space from the buffer cache dict: sdb.open() failed: Cannot allocate memory
I'm beginning to think it was a mistake to ever put Berkeley DB code to Dovecot. SQLite would work just as well (and without problems), right?
Timo Sirainen wrote:
I'm beginning to think it was a mistake to ever put Berkeley DB code to Dovecot. SQLite would work just as well (and without problems), right?
You're probably right. Want to bite the bullet?
-R
On Tue, 2009-06-16 at 20:42 +0200, Ralph Seichter wrote:
Timo Sirainen wrote:
I'm beginning to think it was a mistake to ever put Berkeley DB code to Dovecot. SQLite would work just as well (and without problems), right?
You're probably right. Want to bite the bullet?
The only thing it needs is a trigger similar to PostgreSQL has in http://wiki.dovecot.org/Plugins/Expire.
Timo Sirainen wrote:
The only thing it needs is a trigger similar to PostgreSQL has in http://wiki.dovecot.org/Plugins/Expire.
Well, at least SQLite knows CREATE TRIGGER, which might be a good omen. ;-) Speaking of DB backends for the expire plugin, I experimented with MySQL in combination with Dovecot 1.1.16 as long as BerkeleyDB remains uncooperative. This is the mailbox layout:
john john/Admin/Foo john/Trash
sue sue/News/Bar sue/Trash
I'm using the sieve plugin to divert some incoming mail to john/Admin/Foo or sue/News/Bar respectively, but the 'expires' table in MySQL never shows entries for these two mailboxes. Moving mail manually to these two mailboxes (using Thunderbird) also fails to generate entries in 'expires'. So far, there are only two lines in the 'expires' table:
john/Trash sue/Trash
These were created after I deleted messages using Thunderbird. Have I misconfigured something? My dovecot.conf contains the following data:
protocol imap { mail_plugins = expire }
protocol lda { mail_plugins = cmusieve expire }
dict { expire = mysql:/etc/dovecot/dovecot-dict-expire.conf }
plugin { expire = Trash 1 Trash/* 1 Admin/Foo 3 */Admin/Foo 3 News/Bar 3 */News/Bar 3 */Foo 3 */Bar 3 expire_dict = proxy::expire sieve = /etc/dovecot/sieve/%u.sieve }
As you can see, I tried various patterns to refer to folders Foo and Bar, but it does not seem to work. Trash appears to be OK, though. BTW, these are the contents of /etc/dovecot/dovecot-dict-expire.conf:
connect = host=localhost dbname=db user=dbuser password=dbpass table = expires select_field = expire_stamp where_field = mailbox username_field = not_used
Could you please give me a hint? I am willing to stick with MySQL for the time being, but I have to get it working first. ;-)
-R
On Mon, 2009-06-22 at 14:48 +0200, Ralph Seichter wrote:
expire = Trash 1 Trash/* 1 Admin/Foo 3 */Admin/Foo 3 News/Bar 3 */News/Bar 3 */Foo 3 */Bar 3
What does your namespace configuration look like? Does Dovecot log anything related to expire plugin?
On Tue, 2009-07-07 at 23:21 -0400, Timo Sirainen wrote:
On Mon, 2009-06-22 at 14:48 +0200, Ralph Seichter wrote:
expire = Trash 1 Trash/* 1 Admin/Foo 3 */Admin/Foo 3 News/Bar 3 */News/Bar 3 */Foo 3 */Bar 3What does your namespace configuration look like? Does Dovecot log anything related to expire plugin?
Also v1.2 + http://hg.dovecot.org/dovecot-1.2/rev/df2d4e398c06 could help show more.
Timo Sirainen wrote:
What does your namespace configuration look like? Does Dovecot log anything related to expire plugin?
I have switched to Dovecot 1.2.1 today:
# 1.2.1: /usr/local/dovecot-1.2/etc/dovecot.conf # OS: Linux 2.6.30-gentoo-r2 i686 Gentoo Base System release 2.0.1 protocols: imap login_dir: /usr/local/dovecot-1.2/var/run/dovecot/login login_executable: /usr/local/dovecot-1.2/libexec/dovecot/imap-login mail_location: maildir:~/.maildir mail_debug: yes mail_plugins: expire auth default: mechanisms: plain login passdb: driver: ldap args: /usr/local/dovecot-1.2/etc/dovecot-ldap.conf userdb: driver: passwd socket: type: listen master: path: /usr/local/dovecot-1.2/var/run/dovecot/auth-master mode: 384 plugin: expire: Trash 1 Trash/* 1 Admin/Foo 3 */Admin/Foo 3 News/Bar 3 */News/Bar 3 */Foo 3 */Bar 3 expire_dict: proxy::expire sieve: /usr/local/dovecot-1.2/etc/sieve/%u.sieve dict: expire: mysql:/usr/local/dovecot-1.2/etc/dovecot-dict-expire.conf
The log still does not give me much of a clue. It shows repetitions of
No expiring in mailbox: Admin.Foo No expiring in mailbox: News.Bar
so I think that the slashes in "Admin/Foo" and "News/Bar" etc. are not interpreted correctly. I tried replacing '/' with '.', but it did me no good.
-R
On Fri, 2009-07-10 at 19:27 +0200, Ralph Seichter wrote:
mail_location: maildir:~/.maildir
You have no namespaces configured and you're using maildir..
expire: Trash 1 Trash/* 1 Admin/Foo 3 */Admin/Foo 3 News/Bar 3 */News/Bar 3 */Foo 3 */Bar 3
..
The log still does not give me much of a clue. It shows repetitions of
No expiring in mailbox: Admin.Foo No expiring in mailbox: News.Bar
Right.
so I think that the slashes in "Admin/Foo" and "News/Bar" etc. are not interpreted correctly.
Right, because the separator is '.', not '/' unless you've overridden it in namespace configuration.
I tried replacing '/' with '.', but it did me no good.
What does it log then?
Timo Sirainen wrote:
I tried replacing '/' with '.', but it did me no good.
What does it log then?
"No expiring in mailbox: Admin.Foo", just like before the modification. I don't understand this, because not using namespaces should mean literal string interpretation (or so I thought).
-R
On Fri, 2009-07-10 at 19:40 +0200, Ralph Seichter wrote:
Timo Sirainen wrote:
I tried replacing '/' with '.', but it did me no good.
What does it log then?
"No expiring in mailbox: Admin.Foo", just like before the modification. I don't understand this, because not using namespaces should mean literal string interpretation (or so I thought).
Weird. Can you try what it logs with this patch? http://hg.dovecot.org/dovecot-1.2/rev/bdd8cb7f341a
Timo Sirainen wrote:
"No expiring in mailbox: Admin.Foo", just like before the modification. I don't understand this, because not using namespaces should mean literal string interpretation (or so I thought).
Weird. Can you try what it logs with this patch? http://hg.dovecot.org/dovecot-1.2/rev/bdd8cb7f341a
I started from scratch: Unpacked the original 1.2.1 sources, applied your patch, compiled, installed to a blank directory, wiped the MySQL database table containing expire timestamps. I then copied my existing configuration files to the new Dovecot instance, changed every '/' to '.' in the "expire = ..." line, and now the expire plugin works as expected! This baffles me significantly, because your patch only adds some logging output and I have already tried using '.' before reporting back to you in the first place. I appreciate that it is working now, but it is a weird behaviour nonetheless... (*scratches head*) ?!?
-R
Timo Sirainen schrieb:
On Thu, 2009-06-04 at 22:02 +0200, Ralph Seichter wrote:
Sorry for replying so late, the server running Dovecot succumbed to a hardware problem. Now that the machine is online again, I applied your patch. Running the expire tool now causes the following log messages:
dict: db(secondary, ): DB->set_cachesize: method not permitted when environment specified dict: db(secondary, ): unable to allocate space from the buffer cache dict: sdb.open() failed: Cannot allocate memory
I'm beginning to think it was a mistake to ever put Berkeley DB code to Dovecot. SQLite would work just as well (and without problems), right?
my 2 cent:
Berkeley DB is one of the main reasons why we started to look at (great) alternatives like dovecot instead of cyrus. It's really a pain in the .... with Berkeley. Ask the OpenLDAP guys about the bdb backends and system crashes. Why bother with all the DB_CONFIG and checkpoint and whatever settings, when you can have all comfort of SQL with great speed, simpleness and self recovery of SQLite. The point of self recovery is one of the most great advantages of dovecot, really. BDB is acting opposite to it... :)
And is it a lot easier to get some data out in case of investigations, debugging, backup, etc.
Regards, Sebastian
participants (4)
-
Charles Marcus
-
Ralph Seichter
-
reg9009
-
Timo Sirainen