[Dovecot] Wrong quota calculation

[Cibest ML]

On 09/09/2012 23:30, Daniel Parthey wrote:
> Cibest ML wrote:
>> On 08/09/2012 01:38, Daniel Parthey wrote:
>>> Cibest ML wrote:
>>>> We've a strange quota behavior on our dovecot servers.
>>>> It seems that it's somehow doubling the real used storage (or could
>>>> be something like twice the inbox size + size of other folders, hard
>>>> to know for sure).
>>>> It was initial setup with the maildir++ system. Now it's been
>>>> configured with dict and mysql table but the result is the same.
>> Here are some tests to show the problem. The test mailbox is
>> initialy empty. All described actions are done in a sequencial way
>> (test mail with an attachement of ~2MB)
>>
>> - Message delivered by LMTP
>> Quota name Type    Value  Limit
>> User quota STORAGE  2653 524288
>> User quota MESSAGE     1      -
> Seems ok.
>
>> - quota recalc
>> Quota name Type    Value  Limit
>> User quota STORAGE  5307 524288
>> User quota MESSAGE     2      -
> Strange. There must be some issue with quota recalculation.
> Maybe there are two instances of the message in the maildir?
> How large is the actual maildir measured by diskusage (du)
> command and where are the messages stored in the filesystem?
>
>> - move to draft + compact
>> Quota name Type    Value  Limit
>> User quota STORAGE  2653 524288
>> User quota MESSAGE     1      -
> Seems absolutely correct. Message is copied,
> deleted and removed from the old directory.
>
>> - recalc
>> Quota name Type    Value  Limit
>> User quota STORAGE  2653 524288
>> User quota MESSAGE     1      -
> Seems after compacting the folders there is
> only one instance of the message.
>
>> - move to subfolder "test" in inbox
>> User quota STORAGE  2653 524288
>> User quota MESSAGE     1      -
>> - move message back to inbox
>> Quota name Type    Value  Limit
>> User quota STORAGE  5306 524288
>> User quota MESSAGE     2      -
> The message gets copied and deleted,
> so there are 2 messages.
>
>> - compact
>> Quota name Type    Value  Limit
>> User quota STORAGE  2653 524288
>> User quota MESSAGE     1      -
> After compaction the "copied+deleted"
> message gets finally removed.
>
>> - recalc quota
>> Quota name Type    Value  Limit
>> User quota STORAGE  5306 524288
>> User quota MESSAGE     2      -
> Very strange. The removed message "reappeared"?
> Are you using some network filesystem
> and possibly experience filesystem caching effects?
> In this case a dovecot director would help.
>
> Regards
> Daniel

The storage is indeed done via NFS. There might be some caching effects 
but in that case, the behaviour should occur for all the test cases. But 
it seems only to happen if the mails are stored in INBOX, not in a 
subfolder of INBOX nor another IMAP folder. We've never been able to 
reproduce the issue if the email is not exactly in INBOX.
We've added some options to check if it's a NFS issue (but it shouldn't 
be needed as indexes aren't on a NFS filer) :
map_disable = yes
mail_fsync = always
mail_nfs_index = yes
It doesn't change anything, same behavior.

Here are some test cases to show that there's always 1 mail stored on 
the filesystem

new incoming mail

du -> 8
ls -> -rw------- 1 vmail vmail 3748 Sep 10 16:11 
1347286291.M194531P4329.mspooll01,S=3748,W=3847


quota get

Quota name Type    Value  Limit
User quota STORAGE     3 524288
User quota MESSAGE     1      -

du -> 8
ls -> -rw------- 1 vmail vmail 3748 Sep 10 16:11 
1347286291.M194531P4329.mspooll01,S=3748,W=3847

quota recacl

Quota name Type    Value  Limit
User quota STORAGE     7 524288
User quota MESSAGE     2      -

du -> 8
ls -> -rw------- 1 vmail vmail 3748 Sep 10 16:11 
1347286291.M194531P4329.mspooll01,S=3748,W=3847

it really seems to be connected to the recalc operation.

Moving to dovecot director would require some work but at the moment, we 
can't be sure it'd solve the issue (as we only have one IMAP/POP server 
+ one LMTP server, we shouldn't have issues with stickiness of connexions)

--
Regards
Sébastien