[Dovecot] dict quota problem
hi!
In table I see this:
| papp.tamas@center.hu | 118671994 | 20437 |
du -sm says, the size of the maildir is 154M.
I use xfs. I can't figure out, why is there a difference.
Can you help me out?
Thank you,
tamas
Papp Tamas <tompos@martos.bme.hu> wrote:
In table I see this:
| papp.tamas@center.hu | 118671994 | 20437 |
du -sm says, the size of the maildir is 154M.
I use xfs. I can't figure out, why is there a difference.
Can you help me out?
"du" also accounts for the metadata of the filesystem. With Maildir++ you have many files per directory which causes the directory inode to increase in size. After mails are deleted, the now empty space inside the directory is not reclaimed (at least not in ext2/3/4, I guess, XFS behaves the same) and you see a difference in size, since dovecot only counts the raw size of the mails.
You can test this for yourself: create a directory, place a 100M big file inside and check with "du". Then delete that file, create 10,000 files with 10KB size inside, check with "du", delete the files and check again.
Grüße, Sven.
-- Sigmentation fault. Core dumped.
On 04/20/2011 01:47 PM, Sven Hartge wrote:
"du" also accounts for the metadata of the filesystem. With Maildir++ you have many files per directory which causes the directory inode to increase in size. After mails are deleted, the now empty space inside the directory is not reclaimed (at least not in ext2/3/4, I guess, XFS behaves the same) and you see a difference in size, since dovecot only counts the raw size of the mails.
You can test this for yourself: create a directory, place a 100M big file inside and check with "du". Then delete that file, create 10,000 files with 10KB size inside, check with "du", delete the files and check again.
Thank you for your prompt answer.
Why does dovecot count only the raw size? Does this mean, dict quota is not usable with maildirs? Is this true for sdbox also?
Thank you,
tamas
On Wed, 2011-04-20 at 16:43 +0200, Papp Tamas wrote:
Why does dovecot count only the raw size? Does this mean, dict quota is not usable with maildirs? Is this true for sdbox also?
Dovecot quota counts only message sizes, not including index files and other metadata. It works the same way with dict, Maildir++ and all other quota backends and all mail storage backends.
Papp Tamas <tompos@martos.bme.hu> wrote:
On 04/20/2011 01:47 PM, Sven Hartge wrote:
"du" also accounts for the metadata of the filesystem. With Maildir++ you have many files per directory which causes the directory inode to increase in size. After mails are deleted, the now empty space inside the directory is not reclaimed (at least not in ext2/3/4, I guess, XFS behaves the same) and you see a difference in size, since dovecot only counts the raw size of the mails.
You can test this for yourself: create a directory, place a 100M big file inside and check with "du". Then delete that file, create 10,000 files with 10KB size inside, check with "du", delete the files and check again.
Thank you for your prompt answer.
Why does dovecot count only the raw size? Does this mean, dict quota is not usable with maildirs? Is this true for sdbox also?
In addition to the answer from Timo:
Counting all meta-data is also very unintuitive for the user. Imagine a user with an INBOX containing 200,000 mails. This will result in a _huge_ directory inode for the INBOX folder (about 20MB for XFS). Now he deletes all those mails and expects a quota usage of 0 bytes. But the quota is still at 20MB, because the directory inode has not been reduced by the filesystem.
I guarantee you, you will get numerous support calls.
At least I got them while using filesystem-based quotas.
Grüße, S°
-- Sigmentation fault. Core dumped.
On 04/20/2011 05:26 PM, Sven Hartge wrote:
Papp Tamas<tompos@martos.bme.hu> wrote:
On 04/20/2011 01:47 PM, Sven Hartge wrote:
"du" also accounts for the metadata of the filesystem. With Maildir++ you have many files per directory which causes the directory inode to increase in size. After mails are deleted, the now empty space inside the directory is not reclaimed (at least not in ext2/3/4, I guess, XFS behaves the same) and you see a difference in size, since dovecot only counts the raw size of the mails.
You can test this for yourself: create a directory, place a 100M big file inside and check with "du". Then delete that file, create 10,000 files with 10KB size inside, check with "du", delete the files and check again. Thank you for your prompt answer. Why does dovecot count only the raw size? Does this mean, dict quota is not usable with maildirs? Is this true for sdbox also? In addition to the answer from Timo:
Counting all meta-data is also very unintuitive for the user. Imagine a user with an INBOX containing 200,000 mails. This will result in a _huge_ directory inode for the INBOX folder (about 20MB for XFS). Now he deletes all those mails and expects a quota usage of 0 bytes. But the quota is still at 20MB, because the directory inode has not been reduced by the filesystem.
I guarantee you, you will get numerous support calls.
At least I got them while using filesystem-based quotas.
I'm sorry for the late response.
So, there is no exact or something closer solution with normal maildirs?
Thank you,
tamas
Papp Tamas <tompos@martos.bme.hu> wrote:
On 04/20/2011 05:26 PM, Sven Hartge wrote:
Counting all meta-data is also very unintuitive for the user. Imagine a user with an INBOX containing 200,000 mails. This will result in a _huge_ directory inode for the INBOX folder (about 20MB for XFS). Now he deletes all those mails and expects a quota usage of 0 bytes. But the quota is still at 20MB, because the directory inode has not been reduced by the filesystem.
I guarantee you, you will get numerous support calls.
At least I got them while using filesystem-based quotas.
So, there is no exact or something closer solution with normal maildirs?
Define "exact".
Trust me, you don't want to include any meta data or filesystem overhead in the quota value presented to the user. Doing so is just asking for trouble and grief.
Grüße, Sven.
-- Sigmentation fault. Core dumped.
On 04/30/2011 01:36 AM, Sven Hartge wrote:
So, there is no exact or something closer solution with normal maildirs? Define "exact".
I mean, how I can set a true quota usage for a user?
Trust me, you don't want to include any meta data or filesystem overhead in the quota value presented to the user. Doing so is just asking for trouble and grief.
I unserstand the overhead, I'm just looking for any kind of solution, which works.
Thank you,
tamas
ps: I think mailman suppose to inject the 'Reply-To: Dovecot Mailing List <dovecot@dovecot.org>' header, doesn't it?
Papp Tamas <tompos@martos.bme.hu> wrote:
On 04/30/2011 01:36 AM, Sven Hartge wrote:
So, there is no exact or something closer solution with normal maildirs?
Define "exact".
I mean, how I can set a true quota usage for a user?
Define "true quota usage".
Trust me, you don't want to include any meta data or filesystem overhead in the quota value presented to the user. Doing so is just asking for trouble and grief.
I unserstand the overhead, I'm just looking for any kind of solution, which works.
The current implementation just works. Where is your problem with that? You get the exact amount of bytes all mails of a user are allocating on the storage (minus the space for the meta data of dovecot like indexes and overhead from the filesystem).
S°
-- Sigmentation fault. Core dumped.
On 04/30/2011 02:24 PM, Sven Hartge wrote:
I mean, how I can set a true quota usage for a user? Define "true quota usage".
Maximum usable space for a user on the disk.
The current implementation just works. Where is your problem with that? You get the exact amount of bytes all mails of a user are allocating on the storage (minus the space for the meta data of dovecot like indexes and overhead from the filesystem).
As I wrote, It does not work. The counted and the real space usage is far away from each other if you have a lot of mail files.
Thank you,
tamas
Papp Tamas <tompos@martos.bme.hu> wrote:
On 04/30/2011 02:24 PM, Sven Hartge wrote:
I mean, how I can set a true quota usage for a user? Define "true quota usage".
Maximum usable space for a user on the disk.
The current implementation just works. Where is your problem with that? You get the exact amount of bytes all mails of a user are allocating on the storage (minus the space for the meta data of dovecot like indexes and overhead from the filesystem).
As I wrote, It does not work. The counted and the real space usage is far away from each other if you have a lot of mail files.
Yes. You as admin have to deal with this. Buy a bigger disk, reduce the quota, etc.
Or use a filesystem quota, which will (of course) only work with real users and not with virtual users.
Grüße, Sven.
-- Sigmentation fault. Core dumped.
On 04/30/2011 10:21 PM, Sven Hartge wrote:
Yes. You as admin have to deal with this. Buy a bigger disk, reduce the quota, etc.
Or use a filesystem quota, which will (of course) only work with real users and not with virtual users.
OK, thank you. Well, virtual users are needed:)
tamas
participants (3)
-
Papp Tamas
-
Sven Hartge
-
Timo Sirainen