[Dovecot] Dovecot usage values differ from actual disk usage
Hi,
in my dovecot setup the accounting database table shows wrong values which do not correspond with the actual disk space used.
The disk usage says 2.6 Gigabytes:
mail01:~# du -sh /mail/dovecot/example.org/username 2.6G /mail/dovecot/example.org/username
While the doveadm quota get says 7 Gigabytes for the same user:
mail01:~# doveadm -f tab quota get -u username@example.org Quota name Type Value Limit % User quota STORAGE 7065208 - 0 User quota MESSAGE 282024 - 0
mysql> select * from dovecot.dovecot_usage where username="username@example.org"; +----------------------+------------+----------+ | username | storage | messages | +----------------------+------------+----------+ | username@example.org | 7234773131 | 282024 | +----------------------+------------+----------+ 1 row in set (0.00 sec)
What might be the reason for such huge differences (several gigabytes) between storage value in dovecot_usage table and disk usage value?
After a quota recalc everything is correct:
mail01:~# doveadm quota recalc -u username@example.org mail01:~# doveadm -f tab quota get -u username@example.org Quota name Type Value Limit % User quota STORAGE 2630434 - 0 User quota MESSAGE 100714 - 0
mysql> select * from dovecot.dovecot_usage where username="username@example.org"; +----------------------+------------+----------+ | username | storage | messages | +----------------------+------------+----------+ | username@example.org | 2693565239 | 100714 | +----------------------+------------+----------+ 1 row in set (0.00 sec)
How can I keep the accounting data consistent with actual disk usage, without running a recalc cronjob every few hours?
There's running a dovecot director and mailbox instance on each of our servers with the mails being stored on NFS and received from postfix via LMTP through the director lmtp port 20024.
Configuration of director (dovecot-director.conf) and mailbox (dovecot.conf) is attached.
Kind regards Daniel
On Sun, 2012-05-13 at 02:07 +0200, Daniel Parthey wrote:
in my dovecot setup the accounting database table shows wrong values which do not correspond with the actual disk space used.
Can you easily reproduce this with a test user? How?
If there's no easy way to manually reproduce this, try if you can reproduce it by running imaptest against a test user: http://imapwiki.org/ImapTest
In the tests does it make difference if you bypass director?
Hi Timo,
Timo Sirainen wrote:
On Sun, 2012-05-13 at 02:07 +0200, Daniel Parthey wrote:
in my dovecot setup the accounting database table shows wrong values which do not correspond with the actual disk space used.
Can you easily reproduce this with a test user? How?
Maybe this is not the original problem, but here is some way to reproduce a similar behaviour.
##################### # Close Thunderbird # #####################
################# # Clear Mailbox # #################
rm -rf /mail/dovecot/example.org/user1 sleep 1 doveadm quota recalc -u user1@example.org
############### # Check quota # ###############
# doveadm quota get -u user1@example.org Quota name Type Value Limit % User quota STORAGE 0 - 0 User quota MESSAGE 0 - 0
################ # Deliver mail # ################
{
cat < ###############
# Check quota #
############### # doveadm quota get -u user1@example.org
Quota name Type Value Limit %
User quota STORAGE 1383 - 0
User quota MESSAGE 1 - 0 #######################
# Check mdbox storage #
####################### du -sk /mail/dovecot/example.org/user1
1432 /mail/dovecot/example.org/user1 #####################
# Move Email around #
##################### Move Email two times between Trash and INBOX (4 moves in total).
There is still only one physical message, just moved it around a bit. #####################
# Check quota again #
##################### # doveadm quota get -u user1@example.org
Quota name Type Value Limit %
User quota STORAGE 6918 - 0
User quota MESSAGE 5 - 0 Quota now displays 5 messages and 6 MB, even though there is only one message with 1.5 MB. #######################
# Check mdbox storage #
####################### # du -sk /mail/dovecot/example.org/user1
1472 /mail/dovecot/example.org/user1 The actual disk usage did not increase, but the "usage value" did increase
with every move operation. As long as the client has its MUA open, the usage is incorrectly displayed
(and accounted), and does not correspond with actual disk usage. #####################
# Close Thunderbird #
##################### The mailboxes get expunged. #####################
# Check quota again #
##################### # doveadm quota get -u user1@example.org
Quota name Type Value Limit %
User quota STORAGE 1383 - 0
User quota MESSAGE 1 - 0 Usage value and message count is correct again, once thunderbird is closed. #######################
# Check mdbox storage #
####################### # du -sk /mail/dovecot/example.org/user1
1472 /mail/dovecot/example.org/user1 The actual disk usage did not change, which is OK. In the tests does it make difference if you bypass director? No. The behaviour is the same if I access the mailbox directly. Regards,
Daniel
On Thu, 2012-05-17 at 20:11 +0200, Daniel Parthey wrote:
Maybe this is not the original problem, but here is some way to reproduce a similar behaviour.
This behavior is intentional.
##################### # Move Email around # #####################
Move Email two times between Trash and INBOX (4 moves in total). There is still only one physical message, just moved it around a bit.
You're not moving a message, you're copying and setting \deleted flag for the original message. Until client issues EXPUNGE command, the messages exist.
##################### # Check quota again # #####################
# doveadm quota get -u user1@example.org Quota name Type Value Limit % User quota STORAGE 6918 - 0 User quota MESSAGE 5 - 0
Quota now displays 5 messages and 6 MB, even though there is only one message with 1.5 MB.
####################### # Check mdbox storage # #######################
# du -sk /mail/dovecot/example.org/user1 1472 /mail/dovecot/example.org/user1
The actual disk usage did not increase, but the "usage value" did increase with every move operation.
The message still has 5 visible instances, even though in mdbox they don't use up any more disk space than 1. It would be nice for quota plugin to realize this, but it's a big and probably difficult change. I've thought about it once in a while though.
Timo Sirainen wrote:
This behavior is intentional.
Move Email two times between Trash and INBOX (4 moves in total). There is still only one physical message, just moved it around a bit.
You're not moving a message, you're copying and setting \deleted flag for the original message. Until client issues EXPUNGE command, the messages exist.
OK.
The message still has 5 visible instances, even though in mdbox they don't use up any more disk space than 1. It would be nice for quota plugin to realize this, but it's a big and probably difficult change. I've thought about it once in a while though.
Maybe I should have a deeper look into how to reproduce this. Originally, I did an imap-sync[1] from dbmail to dovecot through the director IMAP port. After some days of mailbox usage in dovecot the quota values were obviously incorrect from the administrators view, but I'm not sure whether the user kept his mailbox and IMAP connection open the whole week. I will try to find this out.
Regards Daniel
Daniel Parthey wrote:
Timo Sirainen wrote:
The message still has 5 visible instances, even though in mdbox they don't use up any more disk space than 1. It would be nice for quota plugin to realize this, but it's a big and probably difficult change. I've thought about it once in a while though.
Maybe I should have a deeper look into how to reproduce this. Originally, I did an imap-sync[1] from dbmail to dovecot through the director IMAP port. After some days of mailbox usage in dovecot the quota values were obviously incorrect from the administrators view, but I'm not sure whether the user kept his mailbox and IMAP connection open the whole week. I will try to find this out.
I now repeated a fresh imap-sync from a dbmail mailbox containing 60216 mails and dovecot displays exactly 60216 mails, this seems correct:
mail01:~# doveadm quota get -u user1@example.org; du -sb /mail/dovecot/example.org/user1/mail Quota name Type Value Limit % User quota STORAGE 1252372 - 0 User quota MESSAGE 60216 - 0 1338670280 /mail/dovecot/example.org/user1/mail
I assume the reason for the wrong quota values was the following:
The user has already had a dovecot test account before his mailbox has been migrated from dbmail to dovecot. If you clear/remove a mailbox using rm -rf, the storage values in database are not touched. And then the storage/quota_usage becomes wrong if you don't do a doveadm quota recalc for this user after moving his mail folder out of the way.
Any additional imported/incoming mails will only increase the storage values, but not the size of the mails which have been deletedv from the filesystem.
Kind regards, Daniel
participants (2)
-
Daniel Parthey
-
Timo Sirainen