[Dovecot] mdbox expunge purge question
Hi,
I'm testing Dovecot 2.0rc2 with mdbox mail format and I'm not sure what this exactly means:
http://wiki2.dovecot.org/MailboxFormat/dbox
Quote from wiki page:
===== Expunging a message only decreases the message's refcount. The space is later freed in "purge" step. This may be done automatically within the session or later in a nightly cronjob when there's less disk I/O.
So this means that even if a user deletes a mail the space gets only freed if I run "doveadm purge" manually or an user requests this over his mail client?
When the expunge plugin is used the deleted mails are stored in a special mailbox in the users mailbox so it's needed to use the expire plugin too if this should be done automatically?
Is it recommend to run "doveadm purge" periodically if mdbox is used?
Btw: when I try to run
doveadm -Dv expunge -u testuser@dlutt.de mailbox Trash savedbefore 30d
for example I get such an error:
doveadm(root): Debug: Loading modules from directory: /usr/lib/dovecot/modules doveadm(root): Debug: Module loaded: /usr/lib/dovecot/modules/lib01_acl_plugin.so doveadm(root): Debug: Loading modules from directory: /usr/lib/dovecot/modules/doveadm doveadm(root): Error: dlopen(/usr/lib/dovecot/modules/doveadm/lib10_doveadm_expire_plugin.so) failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_expire_plugin.so: undefined symbol: expire_set_lookup doveadm(root): Error: dlopen(/usr/lib/dovecot/modules/doveadm/lib10_doveadm_quota_plugin.so) failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_quota_plugin.so: undefined symbol: quota_user_module
Plugin expire or expunge is not currently enabled in Dovecot's config.
Thanks for any hints.
Daniel
On Sat, Jul 17, 2010 at 6:29 AM, Daniel Luttermann daniel@dlutt.de wrote:
Hi,
I'm testing Dovecot 2.0rc2 with mdbox mail format and I'm not sure what this exactly means:
http://wiki2.dovecot.org/MailboxFormat/dbox
Quote from wiki page:
===== Expunging a message only decreases the message's refcount. The space is later freed in "purge" step. This may be done automatically within the session or later in a nightly cronjob when there's less disk I/O.
So this means that even if a user deletes a mail the space gets only freed if I run "doveadm purge" manually or an user requests this over his mail client?
When the expunge plugin is used the deleted mails are stored in a special mailbox in the users mailbox so it's needed to use the expire plugin too if this should be done automatically?
Is it recommend to run "doveadm purge" periodically if mdbox is used?
Btw: when I try to run
doveadm -Dv expunge -u testuser@dlutt.de mailbox Trash savedbefore 30d
for example I get such an error:
doveadm(root): Debug: Loading modules from directory: /usr/lib/dovecot/modules doveadm(root): Debug: Module loaded: /usr/lib/dovecot/modules/lib01_acl_plugin.so doveadm(root): Debug: Loading modules from directory: /usr/lib/dovecot/modules/doveadm doveadm(root): Error: dlopen(/usr/lib/dovecot/modules/doveadm/lib10_doveadm_expire_plugin.so) failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_expire_plugin.so: undefined symbol: expire_set_lookup doveadm(root): Error: dlopen(/usr/lib/dovecot/modules/doveadm/lib10_doveadm_quota_plugin.so) failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_quota_plugin.so: undefined symbol: quota_user_module
Plugin expire or expunge is not currently enabled in Dovecot's config.
Thanks for any hints.
Daniel
I was curious about that as well but had already posted a million questions yesterday =P
What is the default mailbox file size (the m* files?), what actually happens when a message is delivered, when is a new file created, when a user deletes a single mail what happens, or if they delete multiple mails, if a users client does the move to trash then mark as deleted then purge, what happens inside? There was mention of no file truncating, so does that mean if a user deletes mails, and they are in 3 different mail storage files, that those messages are written to 3 new (or existing with free space?) mail storage files and the old ones just sit around using space?
And if you never run a purge or expunge script, will your data store just keep growing, is there no self cleanup? With maildir, you delete a message and the file goes away.
I guess if someone had time to write a slightly more detailed explanation of how mdbox works than what is on the wiki that would be cool. Also maybe a "if you run mdbox, you are going to want to know this and that, and do this and this and this"
On 17.7.2010, at 18.55, Brandon Lamb wrote:
What is the default mailbox file size (the m* files?),
mdbox_rotate_size = 2M
what actually happens when a message is delivered, when is a new file created,
It tries to find an existing file to append to, and creates a new one if it can't. The details of this aren't really perfect yet and the code should be rewritten.
when a user deletes a single mail what happens, or if they delete multiple mails,
A few bytes are written to index files.
if a users client does the move to trash then mark as deleted then purge, what happens inside?
That's a COPY + delete. Same as above.
There was mention of no file truncating, so does that mean if a user deletes mails, and they are in 3 different mail storage files, that those messages are written to 3 new (or existing with free space?) mail storage files and the old ones just sit around using space?
doveadm purge goes through all files that have expunged messages. It writes them to other files the exactly the same way as any other new messages are written (so they can be written to other existing files). After it's finished writing them, the old files are deleted.
And if you never run a purge or expunge script, will your data store just keep growing, is there no self cleanup? With maildir, you delete a message and the file goes away.
There is currently no self cleanup. There used to be for a while, but its performance sucked.
I guess if someone had time to write a slightly more detailed explanation of how mdbox works than what is on the wiki that would be cool.
Maybe some day :)
Also maybe a "if you run mdbox, you are going to want to know this and that, and do this and this and this"
Purging is pretty much the only important part. Then there are the couple of mdbox_* settings.
On Sat, Jul 17, 2010 at 11:19 AM, Timo Sirainen tss@iki.fi wrote:
On 17.7.2010, at 18.55, Brandon Lamb wrote:
What is the default mailbox file size (the m* files?),
mdbox_rotate_size = 2M
what actually happens when a message is delivered, when is a new file created,
It tries to find an existing file to append to, and creates a new one if it can't. The details of this aren't really perfect yet and the code should be rewritten.
when a user deletes a single mail what happens, or if they delete multiple mails,
A few bytes are written to index files.
if a users client does the move to trash then mark as deleted then purge, what happens inside?
That's a COPY + delete. Same as above.
There was mention of no file truncating, so does that mean if a user deletes mails, and they are in 3 different mail storage files, that those messages are written to 3 new (or existing with free space?) mail storage files and the old ones just sit around using space?
doveadm purge goes through all files that have expunged messages. It writes them to other files the exactly the same way as any other new messages are written (so they can be written to other existing files). After it's finished writing them, the old files are deleted.
And if you never run a purge or expunge script, will your data store just keep growing, is there no self cleanup? With maildir, you delete a message and the file goes away.
There is currently no self cleanup. There used to be for a while, but its performance sucked.
I guess if someone had time to write a slightly more detailed explanation of how mdbox works than what is on the wiki that would be cool.
Maybe some day :)
Also maybe a "if you run mdbox, you are going to want to know this and that, and do this and this and this"
Purging is pretty much the only important part. Then there are the couple of mdbox_* settings.
Great info thanks, I think that actually answered everything I had questions about!
On 17.7.2010, at 14.29, Daniel Luttermann wrote:
Quote from wiki page:
===== Expunging a message only decreases the message's refcount. The space is later freed in "purge" step. This may be done automatically within the session or later in a nightly cronjob when there's less disk I/O.
There is currently no automatic purge. I updated the wiki.
So this means that even if a user deletes a mail the space gets only freed if I run "doveadm purge" manually
Right.
or an user requests this over his mail client?
User can't request this.
When the expunge plugin is used the deleted mails are stored in a special mailbox in the users mailbox so it's needed to use the expire plugin too if this should be done automatically?
Do you mean lazy_expunge plugin? The expunged mails aren't automatically deleted, so yes, you'll need to delete them. Expire plugin isn't necessary, but it can optimize to skip users who don't have anything to expunge.
Is it recommend to run "doveadm purge" periodically if mdbox is used?
If you don't want to use up all your disk space, yes :)
Btw: when I try to run
doveadm -Dv expunge -u testuser@dlutt.de mailbox Trash savedbefore 30d
doveadm(root): Error: dlopen(/usr/lib/dovecot/modules/doveadm/lib10_doveadm_expire_plugin.so) failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_expire_plugin.so: undefined symbol: expire_set_lookup doveadm(root): Error: dlopen(/usr/lib/dovecot/modules/doveadm/lib10_doveadm_quota_plugin.so) failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_quota_plugin.so: undefined symbol: quota_user_module
Plugin expire or expunge is not currently enabled in Dovecot's config.
That's intentional. They aren't really errors and they're not visible if you don't give -D parameter. doveadm just tries to load all plugins and ignores the ones that it can't (because you haven't enabled expire or quota in mail_plugins).
participants (3)
-
Brandon Lamb
-
Daniel Luttermann
-
Timo Sirainen