[Dovecot] Quota and Trash
Hi,
I finally got a grip on our mail guys to try out Dovecot after they started migrating a ~15k user mailsystem to Maildir+NFS (Netapp)+Courier and the performance was not as good as expected. So far the results are impressive, although there are of course a few small issues to be solved.
One is the current Quota handling, which they use with Courier IMAP and (still) think it needs to be replicated 1:1 to dovecot.
Currently they use maildir++ quota format. Mails moved to Trash get not included in the Quota calculation (standard "feature" of Courier IMAP) and get expunged after a certain period of days (IMAP_EMPTYTRASH).
Is there any way we could replicate this behaviour with Dovecot? Not including mails in the Trash-folder into quota calculation seems to be a no-go to me according to the documentation, you can ignore the quota limit when copying messages into a certain folder with the "ignore" keyword but they are still included.
This issue is directly related with the second one. Deleting old mails in Trash could be archieved with the expire-plugin, but that does not work with Dovecot v1.0, and I'd definitely want to run this. So I have to use "find", which deletes the mail from the folder. However, those mails are still included in the Quota calculation, so that will very likely become incorrect.
Is there a generic recommendation for a tool that could be used to rebuild maildir quota calculations? Any other ideas how I could solve this problem and have dovecot behave like Courier here?
Thanks, Bernhard
Hi Bernhard,
Hi,
I finally got a grip on our mail guys to try out Dovecot after they started migrating a ~15k user mailsystem to Maildir+NFS (Netapp)+Courier and the performance was not as good as expected. So far the results are impressive, although there are of course a few small issues to be solved.
One is the current Quota handling, which they use with Courier IMAP and (still) think it needs to be replicated 1:1 to dovecot.
Bernhard Schmidt schrieb: possible with dovecot lda, at my knowlegde, dovecot lda only misses the quota warn message like maildrop does ( but there is a patch written as far i know ) after all you can use maildrop with dovecot too. i have also experimental sieve server here , so you can use sieve for filtering, and edit with avelsieves squirrel plugin.
Currently they use maildir++ quota format. Mails moved to Trash get not included in the Quota calculation (standard "feature" of Courier IMAP) and get expunged after a certain period of days (IMAP_EMPTYTRASH).
use ignore trash parameter, this should do the job, read the wiki
Is there any way we could replicate this behaviour with Dovecot? Not including mails in the Trash-folder into quota calculation seems to be a no-go to me according to the documentation, you can ignore the quota limit when copying messages into a certain folder with the "ignore" keyword but they are still included.
This issue is directly related with the second one. Deleting old mails in Trash could be archieved with the expire-plugin, but that does not work with Dovecot v1.0, and I'd definitely want to run this. So I have to use "find", which deletes the mail from the folder. However, those mails are still included in the Quota calculation, so that will very likely become incorrect.
perhaps you might try trash plugin, which deletes mail i.e trash folder if the the mailbox would run over quota by incomming mail this seems to me a better koncept, than simply delete mail by their age read the wiki about it.
cant say what you mean by quota calculation, i use quota in mysql , so i have realtime quotas with thunderbird imap
Is there a generic recommendation for a tool that could be used to rebuild maildir quota calculations? Any other ideas how I could solve this problem and have dovecot behave like Courier here?
Thanks, Bernhard
-- Mit freundlichen Gruessen Best Regards
Robert Schetterer
https://www.schetterer.org Munich/Bavaria/Germany
On Mon, Mar 12, 2007 at 06:01:05PM +0100, Robert Schetterer wrote:
Hi Robert,
Currently they use maildir++ quota format. Mails moved to Trash get not included in the Quota calculation (standard "feature" of Courier IMAP) and get expunged after a certain period of days (IMAP_EMPTYTRASH). use ignore trash parameter, this should do the job, read the wiki
It does a similar job, I've read the wiki very thoroughly, but it does not do the same, at least according to the documentation
Courier ignores Trash for Quota calculation, means that when I move a mail from "INBOX" to "Trash" my used quota is decreased.
Dovecot does not ignore the Trash for Quota calculations. When I move (delete) mail the quota is decreased (delete from INBOX), then increased by the same value (append to Trash). The ignore option only allows Dovecot to ignore quota _violations_ when appending to Trash.
This issue is directly related with the second one. Deleting old mails in Trash could be archieved with the expire-plugin, but that does not work with Dovecot v1.0, and I'd definitely want to run this. So I have to use "find", which deletes the mail from the folder. However, those mails are still included in the Quota calculation, so that will very likely become incorrect. perhaps you might try trash plugin, which deletes mail i.e trash folder if the the mailbox would run over quota by incomming mail this seems to me a better koncept, than simply delete mail by their age read the wiki about it.
It is not the same. The mailservers and the storage would explode horribly if all ~15k users would be able to keep their Trash filled until they reach their Quota. Allowing them to reclaim accidentally deleted mails from Trash for three days has proven to be a good solution here.
cant say what you mean by quota calculation, i use quota in mysql , so i have realtime quotas with thunderbird imap
Well, I guess you have the same issue then.
As explained above, when I move a message to Trash in Dovecot it is still counted on the used space in the quota calculations, thus included in the maildirsize. When I remove mails manually now on the filesystem (e.g. using the "find . -mtime ..." listed in the wiki), they get removed from the storage, but since find does not touch maildirsize they are still counted in there. So after some iterations the quota in maildirsize would be used to 100%, while the user would only be using a fraction of it on the disc.
I think the problem appears whenever you use tools on the maildir backend that do not handle quota. Whether the value is stored in maildir++ or dict should not make any difference, as long as the value is stored somewhere. Quota backends like fs or dirsize don't suffer from this, but fs is not usable in our environment and dirsize has a heavy performance penalty.
So having a solution for my first problem (ignoring Trash in Quota calculations, _not_ violation processing) would solve the second, since mails deleted (or not deleted) there are not in quota anymore.
Regards, Bernhard
On Mon, 2007-03-12 at 20:42 +0100, Bernhard Schmidt wrote:
On Mon, Mar 12, 2007 at 06:01:05PM +0100, Robert Schetterer wrote:
Hi Robert,
Currently they use maildir++ quota format. Mails moved to Trash get not included in the Quota calculation (standard "feature" of Courier IMAP) and get expunged after a certain period of days (IMAP_EMPTYTRASH). use ignore trash parameter, this should do the job, read the wiki
It does a similar job, I've read the wiki very thoroughly, but it does not do the same, at least according to the documentation
I think you confused this with the explanation for quota-rewrite. They're different things. I updated the wiki now.
Dovecot does not ignore the Trash for Quota calculations. When I move (delete) mail the quota is decreased (delete from INBOX), then increased by the same value (append to Trash). The ignore option only allows Dovecot to ignore quota _violations_ when appending to Trash.
This is actually a bug. It worked a bit inconsistently. Fixed: http://dovecot.org/list/dovecot-cvs/2007-March/008103.html
On Mon, Mar 12, 2007 at 10:42:45PM +0200, Timo Sirainen wrote:
I think you confused this with the explanation for quota-rewrite. They're different things. I updated the wiki now.
Thanks a lot, looks like it might indeed to what I'm looking for. Robert, Timo, thanks for pointing that out, I don't have access to the testboxes at the moment.
I'll point our mailguys to rc27 as soon as it is out. They're pretty conservative, but so far them using dovecot has worked out well.
Just the last question, is there a way to force recalculation of the currently used quota for maildir++? Just remove the maildirsize file (or, if quota set manually, remove all but the first line of it)?
Thanks, Bernhard
On Mon, 2007-03-12 at 21:54 +0100, Bernhard Schmidt wrote:
Just the last question, is there a way to force recalculation of the currently used quota for maildir++? Just remove the maildirsize file
Yes.
(or, if quota set manually, remove all but the first line of it)?
No. That doesn't seem to work. Hmm. I suppose it should. I'll fix that too.
Timo Sirainen schrieb:
On Mon, 2007-03-12 at 20:42 +0100, Bernhard Schmidt wrote:
On Mon, Mar 12, 2007 at 06:01:05PM +0100, Robert Schetterer wrote:
Hi Robert,
Currently they use maildir++ quota format. Mails moved to Trash get not included in the Quota calculation (standard "feature" of Courier IMAP) and get expunged after a certain period of days (IMAP_EMPTYTRASH). use ignore trash parameter, this should do the job, read the wiki It does a similar job, I've read the wiki very thoroughly, but it does not do the same, at least according to the documentation
I think you confused this with the explanation for quota-rewrite. They're different things. I updated the wiki now.
Dovecot does not ignore the Trash for Quota calculations. When I move (delete) mail the quota is decreased (delete from INBOX), then increased by the same value (append to Trash). The ignore option only allows Dovecot to ignore quota _violations_ when appending to Trash.
This is actually a bug. It worked a bit inconsistently. Fixed: http://dovecot.org/list/dovecot-cvs/2007-March/008103.html
Hi, oehm.....i should read the whole mails on list before i answer *g So Bernhard listen to Timo hi is Lord Jedi of dovecot i didnt noticed that as a bug
Mit freundlichen Gruessen Best Regards
Robert Schetterer
https://www.schetterer.org Munich/Bavaria/Germany
Hi Bernhard,
Bernhard Schmidt schrieb:
On Mon, Mar 12, 2007 at 06:01:05PM +0100, Robert Schetterer wrote:
Hi Robert,
Currently they use maildir++ quota format. Mails moved to Trash get not included in the Quota calculation (standard "feature" of Courier IMAP) and get expunged after a certain period of days (IMAP_EMPTYTRASH). use ignore trash parameter, this should do the job, read the wiki
It does a similar job, I've read the wiki very thoroughly, but it does not do the same, at least according to the documentation
jep your right its not the same to courier, but is see no real problem if using ignore trash and trash plugin together you have a function wich solves the problem too, ok mail in trash get added to your quota, but users would not get into trouble with that, and its not of evil if they mean they got less quota left, it would only be a problem if they mean they still have place to store
Courier ignores Trash for Quota calculation, means that when I move a mail from "INBOX" to "Trash" my used quota is decreased.
Dovecot does not ignore the Trash for Quota calculations. When I move (delete) mail the quota is decreased (delete from INBOX), then increased by the same value (append to Trash). The ignore option only allows Dovecot to ignore quota _violations_ when appending to Trash.
This issue is directly related with the second one. Deleting old mails in Trash could be archieved with the expire-plugin, but that does not work with Dovecot v1.0, and I'd definitely want to run this. So I have to use "find", which deletes the mail from the folder. However, those mails are still included in the Quota calculation, so that will very likely become incorrect. perhaps you might try trash plugin, which deletes mail i.e trash folder if the the mailbox would run over quota by incomming mail this seems to me a better koncept, than simply delete mail by their age read the wiki about it.
It is not the same. The mailservers and the storage would explode horribly if all ~15k users would be able to keep their Trash filled until they reach their Quota. Allowing them to reclaim accidentally deleted mails from Trash for three days has proven to be a good solution here.
youre right, but the koncept of deleting mails from i.e trash as long as place can be made free doesnt brake the whole mailbox quota, cause if there isnt any mail left to delete ( after all the oldest ones got deleted first ), the incomming mail will be bounced with overquota message, youre right this maby result in miracle questions by your users, but this is only a question how to give that faq to them.
cant say what you mean by quota calculation, i use quota in mysql , so i have realtime quotas with thunderbird imap
Well, I guess you have the same issue then.
no i see no problem, works as designed
As explained above, when I move a message to Trash in Dovecot it is still counted on the used space in the quota calculations, thus included in the maildirsize. When I remove mails manually now on the filesystem (e.g. using the "find . -mtime ..." listed in the wiki), they get removed from the storage, but since find does not touch maildirsize they are still counted in there. So after some iterations the quota in maildirsize would be used to 100%, while the user would only be using a fraction of it on the disc.
if you using a script simply rewrite maildirsize too if needed. i use postfix vda , and it does the rewrite if new quota is set in sql with the next incomming mail, dovecot never had a problem with that, after all you will see the new quota in imap after next restart of thunderbird, this issue must be given to users as server faq, and its the same with using maildrop and touching maildirsize out of sql
I think the problem appears whenever you use tools on the maildir backend that do not handle quota. Whether the value is stored in maildir++ or dict should not make any difference, as long as the value is stored somewhere. Quota backends like fs or dirsize don't suffer from this, but fs is not usable in our environment and dirsize has a heavy performance penalty.
So having a solution for my first problem (ignoring Trash in Quota calculations, _not_ violation processing) would solve the second, since mails deleted (or not deleted) there are not in quota anymore.
Regards, Bernhard
i have servers with cyrus, courier and dovecot in production and i have no problems with them and none of our users have, choose the one which fits best to your needs.
-- Mit freundlichen Gruessen Best Regards
Robert Schetterer
https://www.schetterer.org Munich/Bavaria/Germany
participants (3)
-
Bernhard Schmidt
-
Robert Schetterer
-
Timo Sirainen