[Dovecot] quota: ignore deleted messages (?)
Hi!
I am currently in the process of preparing a migration of our old Courier-based IMAP/POP server setup to a Dovecot-based one.
During this process I came across the following problem with the difference Courier and Dovecot handle deleted messages and mail quota.
Quote from http://www.courier-mta.org/imap/README.maildirquota.html:
,---- | The default application configuration that uses this maildirquota | library does not count deleted messages, and any contents of the Trash | folder, against the quota. Messages that are marked as deleted (but not | yet actually removed), or messages that are moved to the Trash folder | (which is subject to automatic purging) do not count towards the set | quota. `----
Ignoring the content (or increasing the quota) of the Trash folder is easy and no problem, but ignoring deleted messages seems impossible without changes to the code.
While deleted messages are still stored on the server and still take up space until they are expunged, counting them against the quota is somewhat counter-intuitive because most clients don't show those mails and the normal user is unaware the mails he deleted are still there and take up space. Worse yet, if the client uses move-to-Trash, a user deleting mails will double the used space (until Expunge is used) and this may push him over his quota and thus cause any new mail delivery to fail.
Unfortunately as our users are used to the courier way of handling the quota this will cause trouble after the migration.
And this setup is run at a University, so I don't have any control over the clients a user uses and the behavior of said clients so I am not able to disable the move-to-Trash feature or force an immediate Expunge after a delete.
So I propose an additional flag for the quota_rule config option to be able to enable a more lax interpretation of the quota enforcement:
quota_rule = *:storage=1G:ignoredeleted quota_rule2 = Trash:storage=+100M
Of course I would then have a nightly cronjob which force-expunges all deleted messages so that users can't store mails infinitely in their mailboxes.
Thanks for your time and Grüße, Sven.
-- Sigmentation fault. Core dumped.
On 24.8.2012, at 15.02, Sven Hartge wrote:
Ignoring the content (or increasing the quota) of the Trash folder is easy and no problem, but ignoring deleted messages seems impossible without changes to the code. .. So I propose an additional flag for the quota_rule config option to be able to enable a more lax interpretation of the quota enforcement:
quota_rule = *:storage=1G:ignoredeleted quota_rule2 = Trash:storage=+100M
This can't be implemented race-condition-free without huge changes to code.
Timo Sirainen tss@iki.fi wrote:
On 24.8.2012, at 15.02, Sven Hartge wrote:
Ignoring the content (or increasing the quota) of the Trash folder is easy and no problem, but ignoring deleted messages seems impossible without changes to the code. .. So I propose an additional flag for the quota_rule config option to be able to enable a more lax interpretation of the quota enforcement:
quota_rule = *:storage=1G:ignoredeleted quota_rule2 = Trash:storage=+100M
This can't be implemented race-condition-free without huge changes to code.
Damn, too bad.
I know for sure either my users or my 1st level support team are going to kill me if I don't find a solution.
Is is possible to forcibly expunge a message at once, directly after a client has marked it as deleted? Kind of the opposite of the lazy_expunge plugin?
Grüße, Sven.
-- Sigmentation fault. Core dumped.
Am 24.08.2012 15:13, schrieb Sven Hartge:
Timo Sirainen tss@iki.fi wrote:
On 24.8.2012, at 15.02, Sven Hartge wrote:
Ignoring the content (or increasing the quota) of the Trash folder is easy and no problem, but ignoring deleted messages seems impossible without changes to the code. .. So I propose an additional flag for the quota_rule config option to be able to enable a more lax interpretation of the quota enforcement:
quota_rule = *:storage=1G:ignoredeleted quota_rule2 = Trash:storage=+100M
This can't be implemented race-condition-free without huge changes to code.
Damn, too bad.
I know for sure either my users or my 1st level support team are going to kill me if I don't find a solution.
Is is possible to forcibly expunge a message at once, directly after a client has marked it as deleted? Kind of the opposite of the lazy_expunge plugin?
Grüße, Sven.
hm perhaps as workaround
http://wiki2.dovecot.org/Plugins/deleted-to-trash and do often http://wiki2.dovecot.org/Plugins/Expire via cron i.e doveadm expunge -A mailbox Trash savedbefore 1h
-- Best Regards MfG Robert Schetterer
Robert Schetterer robert@schetterer.org wrote:
Am 24.08.2012 15:13, schrieb Sven Hartge:
Timo Sirainen tss@iki.fi wrote:
On 24.8.2012, at 15.02, Sven Hartge wrote:
Ignoring the content (or increasing the quota) of the Trash folder is easy and no problem, but ignoring deleted messages seems impossible without changes to the code.
So I propose an additional flag for the quota_rule config option to be able to enable a more lax interpretation of the quota enforcement:
quota_rule = *:storage=1G:ignoredeleted quota_rule2 = Trash:storage=+100M
This can't be implemented race-condition-free without huge changes to code.
Damn, too bad.
I know for sure either my users or my 1st level support team are going to kill me if I don't find a solution.
Is is possible to forcibly expunge a message at once, directly after a client has marked it as deleted? Kind of the opposite of the lazy_expunge plugin?
hm perhaps as workaround
http://wiki2.dovecot.org/Plugins/deleted-to-trash and do often http://wiki2.dovecot.org/Plugins/Expire via cron i.e doveadm expunge -A mailbox Trash savedbefore 1h
I stumbled upon deleted_to_trash 5 minutes ago. This could work, if the code still works with dovecot 2.1.
Grüße, Sven.
-- Sigmentation fault. Core dumped.
Sven Hartge sven@svenhartge.de wrote:
Robert Schetterer robert@schetterer.org wrote:
Am 24.08.2012 15:13, schrieb Sven Hartge:
Is is possible to forcibly expunge a message at once, directly after a client has marked it as deleted? Kind of the opposite of the lazy_expunge plugin?
hm perhaps as workaround
http://wiki2.dovecot.org/Plugins/deleted-to-trash and do often http://wiki2.dovecot.org/Plugins/Expire via cron i.e doveadm expunge -A mailbox Trash savedbefore 1h
I stumbled upon deleted_to_trash 5 minutes ago. This could work, if the code still works with dovecot 2.1.
Nope, does not compile (dovecot-dev headers are installed):
cc
-fPIC -shared -Wall
-I/usr/include/dovecot
-I/usr/include/dovecot/src
-I/usr/include/dovecot/src/lib
-I/usr/include/dovecot/src/lib-storage
-I/usr/include/dovecot/src/lib-mail
-I/usr/include/dovecot/src/lib-imap
-I/usr/include/dovecot/src/lib-index
-DHAVE_CONFIG_H
deleted-to-trash-plugin.c -o lib_deleted_to_trash_plugin.so
deleted-to-trash-plugin.c: In function ‘mailbox_open_or_create’:
deleted-to-trash-plugin.c:79: error: ‘MAILBOX_FLAG_KEEP_RECENT’ undeclared (first use in this function)
deleted-to-trash-plugin.c:79: error: (Each undeclared identifier is reported only once
deleted-to-trash-plugin.c:79: error: for each function it appears in.)
deleted-to-trash-plugin.c: In function ‘copy_deleted_mail_to_trash’:
deleted-to-trash-plugin.c:136: warning: passing argument 1 of ‘mailbox_keywords_unref’ from incompatible pointer type
/usr/include/dovecot/mail-storage.h:612: note: expected ‘struct mail_keywords **’ but argument is of type ‘struct mailbox *’
deleted-to-trash-plugin.c:136: error: too many arguments to function ‘mailbox_keywords_unref’
make: *** [lib_deleted_to_trash_plugin.so] Error 1
Grüße, Sven.
-- Sigmentation fault. Core dumped.
Sven Hartge sven@svenhartge.de wrote:
Nope, does not compile (dovecot-dev headers are installed):
OK, trying to fix this, without having any deeper knowlege of C (anymore):
deleted-to-trash-plugin.c: In function ‘mailbox_open_or_create’: deleted-to-trash-plugin.c:79: error: ‘MAILBOX_FLAG_KEEP_RECENT’ undeclared (first use in this function)
MAILBOX_FLAG_KEEP_RECENT is not present in 2.1, seems to me it was made the default and MAILBOX_FLAG_DROP_RECENT was introduced as its counterpart.
I removed the flag from the call to mailbox_alloc() in
72 static struct mailbox * 73 mailbox_open_or_create(struct mailbox_list *list, const char *name, 74 const char **error_r) 75 { 76 struct mailbox *box; 77 enum mail_error error; 78 79 box = mailbox_alloc(list, name, MAILBOX_FLAG_NO_INDEX_FILES); 80 if (mailbox_open(box) == 0) { 81 *error_r = NULL; 82 return box; 83 } 84
and retried to compile:
cc
-fPIC -shared -Wall
-I/usr/include/dovecot
-I/usr/include/dovecot/src
-I/usr/include/dovecot/src/lib
-I/usr/include/dovecot/src/lib-storage
-I/usr/include/dovecot/src/lib-mail
-I/usr/include/dovecot/src/lib-imap
-I/usr/include/dovecot/src/lib-index
-DHAVE_CONFIG_H
deleted-to-trash-plugin.c -o lib_deleted_to_trash_plugin.so
deleted-to-trash-plugin.c: In function ‘copy_deleted_mail_to_trash’:
deleted-to-trash-plugin.c:135: warning: passing argument 1 of ‘mailbox_keywords_unref’ from incompatible pointer type
/usr/include/dovecot/mail-storage.h:612: note: expected ‘struct mail_keywords **’ but argument is of type ‘struct mailbox *’
deleted-to-trash-plugin.c:135: error: too many arguments to function ‘mailbox_keywords_unref’
make: *** [lib_deleted_to_trash_plugin.so] Error 1
_and_ now I am at the end of my wisdom. Pointer magic in C has always been a dark dark mystery to me (I learned programming in Pascal, Ada95 and later Perl ...).
Help, anybody?
Grüße, Sven.
-- Sigmentation fault. Core dumped.
On 24.8.2012, at 16.13, Sven Hartge wrote:
quota_rule = *:storage=1G:ignoredeleted quota_rule2 = Trash:storage=+100M
This can't be implemented race-condition-free without huge changes to code.
Damn, too bad.
I know for sure either my users or my 1st level support team are going to kill me if I don't find a solution.
How about just disabling the quota enforcing and doing a nightly run of some type of enforcing (sending notification email and/or disabling new mail delivery until user has more quota again)?
Timo Sirainen tss@iki.fi wrote:
On 24.8.2012, at 16.13, Sven Hartge wrote:
quota_rule = *:storage=1G:ignoredeleted quota_rule2 = Trash:storage=+100M
This can't be implemented race-condition-free without huge changes to code.
Damn, too bad.
I know for sure either my users or my 1st level support team are going to kill me if I don't find a solution.
How about just disabling the quota enforcing and doing a nightly run of some type of enforcing (sending notification email and/or disabling new mail delivery until user has more quota again)?
As a last resort, yes. If possible, I'd like to keep the feedback about mailbox size as direct as possible.
Disabling an account only once per night might be acceptable, but the reenabling of the account, once a user has freed some space, has to be instant or I would get constant complains from the users (the ones with the biggest mailboxes being the professors, which can be quite the pain to work with, if they believe they don't get what they think they are entitled to get).
I know, this all sounds a bit "whiny", but I've been working for over 8 years in this position and the harsh reality made me somewhat cautious.
So far, the description of the delete_to_trash plugin sounds promising, because I can already ignore the Trash (or add to the total quota for this folder and do a nightly expunge run for it), if only it would compile for dovecot 2.1.
Grüße, Sven.
-- Sigmentation fault. Core dumped.
On 25.8.2012, at 1.49, Sven Hartge wrote:
How about just disabling the quota enforcing and doing a nightly run of some type of enforcing (sending notification email and/or disabling new mail delivery until user has more quota again)?
As a last resort, yes. If possible, I'd like to keep the feedback about mailbox size as direct as possible.
Disabling an account only once per night might be acceptable, but the reenabling of the account, once a user has freed some space, has to be instant or I would get constant complains from the users (the ones with the biggest mailboxes being the professors, which can be quite the pain to work with, if they believe they don't get what they think they are entitled to get).
You can use quota warning scripts to send warnings and enable account instantly when it goes under 100%.
So far, the description of the delete_to_trash plugin sounds promising, because I can already ignore the Trash (or add to the total quota for this folder and do a nightly expunge run for it), if only it would compile for dovecot 2.1.
I don't like the idea behind deleted_to_trash and won't offer any support for it if I don't have to.
Timo Sirainen tss@iki.fi wrote:
On 25.8.2012, at 1.49, Sven Hartge wrote:
How about just disabling the quota enforcing and doing a nightly run of some type of enforcing (sending notification email and/or disabling new mail delivery until user has more quota again)?
As a last resort, yes. If possible, I'd like to keep the feedback about mailbox size as direct as possible.
Disabling an account only once per night might be acceptable, but the reenabling of the account, once a user has freed some space, has to be instant or I would get constant complains from the users (the ones with the biggest mailboxes being the professors, which can be quite the pain to work with, if they believe they don't get what they think they are entitled to get).
You can use quota warning scripts to send warnings and enable account instantly when it goes under 100%.
Warning the user at 95%, 97% and 99% using the warning scripts is easy and was already configured and is working like a charm.
Disabling the account in a nightly cronjob will be easy as well. Since I use the Mysql-dict for quota, I can just query that, compare the value to the configured quota from LDAP and act accordingly.
But how do I instantly reenable the account when it drops below 100% with the warning scripts? As far as I understand the documentation, they are only triggered if the quota use rises over the configured thresholds.
I seem to somehow miss a piece, but I cannot see which one.
Grüße, Sven.
-- Sigmentation fault. Core dumped.
On 25.8.2012, at 15.40, Sven Hartge wrote:
Warning the user at 95%, 97% and 99% using the warning scripts is easy and was already configured and is working like a charm.
But how do I instantly reenable the account when it drops below 100% with the warning scripts? As far as I understand the documentation, they are only triggered if the quota use rises over the configured thresholds.
Setting the warning at -100% would do that I think. In any case "-" before the percentage means "reverse". Yeah, not in wiki, I'll add it there..
Timo Sirainen tss@iki.fi wrote:
On 25.8.2012, at 15.40, Sven Hartge wrote:
Warning the user at 95%, 97% and 99% using the warning scripts is easy and was already configured and is working like a charm.
But how do I instantly reenable the account when it drops below 100% with the warning scripts? As far as I understand the documentation, they are only triggered if the quota use rises over the configured thresholds.
Setting the warning at -100% would do that I think. In any case "-" before the percentage means "reverse". Yeah, not in wiki, I'll add it there..
Ah, very nice.
Now I need to put the pieces together and test whether this is a working solution for me.
Question: is it possible to enforce the quota for IMAP/POP and have it on noenforcing for LDA/LMTP?
LDA would be easy, as I could just add -o "plugin/quota=dict:Benutzer-Quota::noenforcing:proxy::quota" to the command line of dovecot-ldap in my exim4.conf.
But because of performance considerations I want to use LMTP to deliver the mails. Would this nested plugin configuration work: (sorry, not at work, cannot test directly)
protocol lmtp { mail_plugins = $mail_plugins sieve plugin { quota = dict:Benutzer-Quota::noenforcing:proxy::quota } }
Grüße, Sven.
-- Sigmentation fault. Core dumped.
On 25.8.2012, at 18.28, Sven Hartge wrote:
But because of performance considerations I want to use LMTP to deliver the mails. Would this nested plugin configuration work: (sorry, not at work, cannot test directly)
protocol lmtp { mail_plugins = $mail_plugins sieve plugin { quota = dict:Benutzer-Quota::noenforcing:proxy::quota } }
That should work, yes.
Timo Sirainen tss@iki.fi wrote:
On 25.8.2012, at 18.28, Sven Hartge wrote:
But because of performance considerations I want to use LMTP to deliver the mails. Would this nested plugin configuration work: (sorry, not at work, cannot test directly)
protocol lmtp { mail_plugins = $mail_plugins sieve plugin { quota = dict:Benutzer-Quota::noenforcing:proxy::quota } }
That should work, yes.
Very good, thank you for your fast answers.
Grüße, Sven.
-- Sigmentation fault. Core dumped.
Sven Hartge wrote:
I want to use LMTP to deliver the mails. Would this nested plugin configuration work to disable quota:
protocol lmtp { mail_plugins = $mail_plugins sieve plugin { quota = dict:Benutzer-Quota::noenforcing:proxy::quota } }
Concerning http://dovecot.org/list/dovecot/2012-June/066298.html
I'm using MySQL userdb, but just want to keep things as simple as possible (KISS principle).
What would the configuration look like if I want to setup one lmtp service with quota enforcing on port 10024 and another lmtp service without enforcing on port 20024? Especially how to define two "protocol lmtp" sections which are assigned to their specific service?
Will this work with two simple 'nested configuration' or will I need to define SQL statements in the userdb lookup which selects "noenforcing" depending on local port?
Regards Daniel
On 26.8.2012, at 2.28, Daniel Parthey wrote:
What would the configuration look like if I want to setup one lmtp service with quota enforcing on port 10024 and another lmtp service without enforcing on port 20024? Especially how to define two "protocol lmtp" sections which are assigned to their specific service?
You can't. (Maybe you could create another service lmtp2 {} block and override the quota setting with lmtp -o plugin/quota=.. parameter, but I don't know if that would work.)
or will I need to define SQL statements in the userdb lookup which selects "noenforcing" depending on local port?
That would work.
El 25/08/12 09:23, Timo Sirainen escribió:
On 25.8.2012, at 1.49, Sven Hartge wrote:
How about just disabling the quota enforcing and doing a nightly run of some type of enforcing (sending notification email and/or disabling new mail delivery until user has more quota again)?
As a last resort, yes. If possible, I'd like to keep the feedback about mailbox size as direct as possible.
Disabling an account only once per night might be acceptable, but the reenabling of the account, once a user has freed some space, has to be instant or I would get constant complains from the users (the ones with the biggest mailboxes being the professors, which can be quite the pain to work with, if they believe they don't get what they think they are entitled to get).
You can use quota warning scripts to send warnings and enable account instantly when it goes under 100%.
But, if you still use quota, you are still counting deleted messages in the quota. Aren't you?
Is not easier just making users conscious of deleted messages? Many clients that hide deleted messages, show them in a trash folder (although they don't use move to trash) so it's easy for user to expunge deleted messages. And you could also run a cron to expunge old deleted messages from user mailboxes (for example, messages deleted more than 15 days ago)
-- Angel L. Mateo Martínez Sección de Telemática Área de Tecnologías de la Información y las Comunicaciones Aplicadas (ATICA) http://www.um.es/atica Tfo: 868887590 Fax: 868888337
Angel L. Mateo amateo@um.es wrote:
El 25/08/12 09:23, Timo Sirainen escribió:
On 25.8.2012, at 1.49, Sven Hartge wrote:
How about just disabling the quota enforcing and doing a nightly run of some type of enforcing (sending notification email and/or disabling new mail delivery until user has more quota again)?
As a last resort, yes. If possible, I'd like to keep the feedback about mailbox size as direct as possible.
Disabling an account only once per night might be acceptable, but the reenabling of the account, once a user has freed some space, has to be instant or I would get constant complains from the users (the ones with the biggest mailboxes being the professors, which can be quite the pain to work with, if they believe they don't get what they think they are entitled to get).
You can use quota warning scripts to send warnings and enable account instantly when it goes under 100%.
But, if you still use quota, you are still counting deleted messages in the quota. Aren't you?
Quota would be noenforcing. So the users still sees "105% of quota used" but this does not result in any mail bouncing at once.
Grüße, Sven.
-- Sigmentation fault. Core dumped.
participants (5)
-
Angel L. Mateo
-
Daniel Parthey
-
Robert Schetterer
-
Sven Hartge
-
Timo Sirainen