[BUG?] Double quota calulation when special folder is present

[Mark Moseley]

On Wed, Apr 3, 2019 at 9:37 PM Mark Moseley <moseleymark at gmail.com> wrote:

>
> On Wed, Mar 20, 2019 at 2:13 PM Mark Moseley <moseleymark at gmail.com>
> wrote:
>
>> Just hoping to get some dev eyes on this. I'm incredibly reluctant to
>> throw the word 'bug' around
>> (since 99 times out of 100, it's not -- it's almost always the config),
>> but I can't think of any way
>> that this could be a config issue, esp when the pre-2.2.34 version works
>> as expected.
>>
>> I noticed during troubleshooting that dovecot errors out if I try to
>> create a subfolder called
>> 'INBOX' but it'll happily create a subfolder called INBOX.SomethingElse
>> (i.e. a folder called
>> INBOX.INBOX.SomethingElse - resulting in a directory called
>> .INBOX.SomethingElse on the
>> filesystem, and leading to the problem described below). Is that
>> sub-subfolder creation (where
>> the top level subfolder matches the namespace name) supposed to be
>> allowed? It seems
>> odd that 'INBOX' (as a subfolder of INBOX) would be blocked but
>> INBOX.SomethingElse (as
>> a subfolder of INBOX) would be allowed. I'd expect INBOX.SomethingElse
>> (i.e.
>> INBOX.INBOX.SomethingElse) would be blocked as well.
>>
>>
>> On Wed, Mar 13, 2019 at 4:46 AM Bernd Wurst via dovecot <
>> dovecot at dovecot.org> wrote:
>>
>>> Hello,
>>>
>>> we're operating dovecot on a small server. Some years ago, we migrated
>>> from courier IMAP to dovecot. Therefore, we defined our default
>>> Namespace "inbox" with prefix "INBOX." to have this compatible. I found
>>> this in some migration docs those days. Generally, everything worked as
>>> expected.
>>>
>>> Our only namespace is configured like this:
>>>
>>> namespace inbox {
>>>  separator = .
>>>   prefix = INBOX.
>>>   inbox = yes
>>> }
>>>
>>> Regularly, there is no folder named INBOX or .INBOX in the file system,
>>> I suppose this is correct.  But I found a special corner case today when
>>> it comes to quota calculation.
>>>
>>> When - for whatever reason - a folder .INBOX.foo (for arbitrary values
>>> of foo) exists, the whole mailbox is counted twice in quota
>>> recalculation. Just creating .INBOX does nothing but a subfolder
>>> triggers the problem.
>>>
>>> This is my shell view (replaced username and file path and deleted
>>> unnecessary debug output)
>>>
>>> $ cat maildirsize
>>> 268435456S
>>> 14697 17
>>> $ maildirmake .INBOX.foo
>>> $ sudo doveadm -D quota recalc -u <user>
>>> [...]
>>> doveadm(<user>): Debug: Namespace inbox: type=private, prefix=INBOX.,
>>> sep=., inbox=yes, hidden=no, list=yes, subscriptions=yes
>>> location=maildir:/home/.../test
>>> doveadm(<user>): Debug: maildir++: root=/home/.../test, index=,
>>> indexpvt=, control=, inbox=/home/.../test, alt=
>>> doveadm(<user>): Debug: Namespace : type=private, prefix=, sep=,
>>> inbox=no, hidden=yes, list=no, subscriptions=no
>>> location=fail::LAYOUT=none
>>> doveadm(<user>): Debug: none: root=, index=, indexpvt=, control=,
>>> inbox=, alt=
>>> doveadm(<user>): Debug: quota: quota_over_flag check: quota_over_script
>>> unset - skipping
>>> doveadm(<user>): Debug: Quota root User quota: Recalculated relative
>>> rules with bytes=268435456 count=0. Now grace=26843545
>>> doveadm(<user>): Debug: Namespace INBOX.: Using permissions from
>>> /home/.../test: mode=0700 gid=default
>>>
>>> $ cat maildirsize
>>> 268435456S
>>> 29394 34
>>>
>>>
>>> So the used quota has exactly been doubled by just creating an empty
>>> subfolder.
>>>
>>> Do you have any pointers for fixing my configuration or is this a bug in
>>> dovecot?
>>>
>>>
>> I coincidentally resurrected a months-old thread with this same issue a
>> few days ago. I'm seeing the exact same after upgrading from 2.2.32 to
>> 2.2.36.
>>
>> The original poster (who also narrowed it down to something in 2.2.34)
>> mentioned a workaround that does indeed work, namely setting
>> mailbox_list_index=no:
>>
>> > doveadm -o 'mailbox_list_index=no' quota recalc -u myuser
>>
>> I've been staring at diffs of 2.2.33 and 2.2.34 without anything jumping
>> out at me (not a C guy, sadly). Maybe src/lib-storage/index/index-storage.c
>> or src/lib-storage/list/mailbox-list-fs-iter.c or
>> src/lib-storage/list/mailbox-list-index-iter.c
>> or src/lib-storage/list/mailbox-list-index.c?
>>
>> The latter few have some added strcmp's against "INBOX". Then again,
>> there's a lot of new code in the diffs under src/lib-storage that
>> references INBOX specifically.
>>
>
> Can the Dovecot team confirm whether this is indeed a bug or not?  I've
> not yet been able to test 2.3.x to see if the problem exists there as well.
>

I've bisected this down to this commit:

git diff
7620195ceeea805137cbd1bae104e385eee474a9..97473a513feb2bbd763051869c8b7b83e24b37fa

diff --git a/src/lib-storage/list/mailbox-list-index-iter.c
b/src/lib-storage/list/mailbox-list-index-iter.c
index c9afc7a..49cd941 100644
--- a/src/lib-storage/list/mailbox-list-index-iter.c
+++ b/src/lib-storage/list/mailbox-list-index-iter.c
@@ -90,13 +90,18 @@ mailbox_list_index_update_info(struct
mailbox_list_index_iterate_context *ctx)
        if (strcmp(ctx->info.vname, "INBOX") != 0) {
                /* non-INBOX */
                ctx->info.vname = p_strdup(ctx->info_pool, ctx->info.vname);
-       } else {
+       } else if (!ctx->prefix_inbox_list) {
                /* listing INBOX itself */
                ctx->info.vname = "INBOX";
                if (mail_namespace_is_inbox_noinferiors(ctx->info.ns)) {
                        ctx->info.flags &=
~(MAILBOX_CHILDREN|MAILBOX_NOCHILDREN);
                        ctx->info.flags |= MAILBOX_NOINFERIORS;
                }
+       } else {
+               /* listing INBOX/INBOX */
+               ctx->info.vname = p_strconcat(ctx->info_pool,
+                       ctx->ctx.list->ns->prefix, "INBOX", NULL);
+               ctx->info.flags |= MAILBOX_NONEXISTENT;
        }
        if ((node->flags & MAILBOX_LIST_INDEX_FLAG_NONEXISTENT) != 0)
                ctx->info.flags |= MAILBOX_NONEXISTENT;
@@ -126,6 +131,14 @@ mailbox_list_index_update_next(struct
mailbox_list_index_iterate_context *ctx,
 {
        struct mailbox_list_index_node *node = ctx->next_node;

+       if (!ctx->prefix_inbox_list && ctx->ctx.list->ns->prefix_len > 0 &&
+           strcmp(node->name, "INBOX") == 0 && node->parent == NULL &&
+           node->children != NULL) {
+               /* prefix/INBOX has children */
+               ctx->prefix_inbox_list = TRUE;
+               return;
+       }
+
        if (node->children != NULL && follow_children) {
                ctx->parent_len = str_len(ctx->path);
                ctx->next_node = node->children;
diff --git a/src/lib-storage/list/mailbox-list-index.h
b/src/lib-storage/list/mailbox-list-index.h
index 9dc1c48..e4b042a 100644
--- a/src/lib-storage/list/mailbox-list-index.h
+++ b/src/lib-storage/list/mailbox-list-index.h
@@ -138,6 +138,7 @@ struct mailbox_list_index_iterate_context {
        struct mailbox_list_index_node *next_node;

        unsigned int failed:1;
+       unsigned int prefix_inbox_list:1;
 };

 extern MODULE_CONTEXT_DEFINE(mailbox_list_index_module,


Prior to this commit, anything updating the quota would do the right thing
for any .INBOX.<something> folders (i.e. not double count the contents of
"INBOX" against the quota). After this commit, anything updating quota (new
mail, quota recalc, etc) does the double counting of INBOX.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://dovecot.org/pipermail/dovecot/attachments/20190409/65a7cc6c/attachment-0001.html>