[BUG?] Double quota calulation when special folder is present

[Mark Moseley]

On Tue, Apr 9, 2019 at 9:52 PM Aki Tuomi <aki.tuomi at open-xchange.com> wrote:

>
> On 10 April 2019 05:00 Mark Moseley via dovecot <dovecot at dovecot.org>
> wrote:
>
>
> On Wed, Apr 3, 2019 at 9:37 PM Mark Moseley < moseleymark at gmail.com>
> wrote:
>
>
> On Wed, Mar 20, 2019 at 2:13 PM Mark Moseley < moseleymark at gmail.com>
> wrote:
>
> Just hoping to get some dev eyes on this. I'm incredibly reluctant to
> throw the word 'bug' around
> (since 99 times out of 100, it's not -- it's almost always the config),
> but I can't think of any way
> that this could be a config issue, esp when the pre-2.2.34 version works
> as expected.
>
> I noticed during troubleshooting that dovecot errors out if I try to
> create a subfolder called
> 'INBOX' but it'll happily create a subfolder called INBOX.SomethingElse
> (i.e. a folder called
> INBOX.INBOX.SomethingElse - resulting in a directory called
> .INBOX.SomethingElse on the
> filesystem, and leading to the problem described below). Is that
> sub-subfolder creation (where
> the top level subfolder matches the namespace name) supposed to be
> allowed? It seems
> odd that 'INBOX' (as a subfolder of INBOX) would be blocked but
> INBOX.SomethingElse (as
> a subfolder of INBOX) would be allowed. I'd expect INBOX.SomethingElse
> (i.e.
> INBOX.INBOX.SomethingElse) would be blocked as well.
>
>
> On Wed, Mar 13, 2019 at 4:46 AM Bernd Wurst via dovecot <
> dovecot at dovecot.org> wrote:
>
> Hello,
>
> we're operating dovecot on a small server. Some years ago, we migrated
> from courier IMAP to dovecot. Therefore, we defined our default
> Namespace "inbox" with prefix "INBOX." to have this compatible. I found
> this in some migration docs those days. Generally, everything worked as
> expected.
>
> Our only namespace is configured like this:
>
> namespace inbox {
>  separator = .
>   prefix = INBOX.
>   inbox = yes
> }
>
> Regularly, there is no folder named INBOX or .INBOX in the file system,
> I suppose this is correct.  But I found a special corner case today when
> it comes to quota calculation.
>
> When - for whatever reason - a folder .INBOX.foo (for arbitrary values
> of foo) exists, the whole mailbox is counted twice in quota
> recalculation. Just creating .INBOX does nothing but a subfolder
> triggers the problem.
>
> This is my shell view (replaced username and file path and deleted
> unnecessary debug output)
>
> $ cat maildirsize
> 268435456S
> 14697 17
> $ maildirmake .INBOX.foo
> $ sudo doveadm -D quota recalc -u <user>
> [...]
> doveadm(<user>): Debug: Namespace inbox: type=private, prefix=INBOX.,
> sep=., inbox=yes, hidden=no, list=yes, subscriptions=yes
> location=maildir:/home/.../test
> doveadm(<user>): Debug: maildir++: root=/home/.../test, index=,
> indexpvt=, control=, inbox=/home/.../test, alt=
> doveadm(<user>): Debug: Namespace : type=private, prefix=, sep=,
> inbox=no, hidden=yes, list=no, subscriptions=no location=fail::LAYOUT=none
> doveadm(<user>): Debug: none: root=, index=, indexpvt=, control=,
> inbox=, alt=
> doveadm(<user>): Debug: quota: quota_over_flag check: quota_over_script
> unset - skipping
> doveadm(<user>): Debug: Quota root User quota: Recalculated relative
> rules with bytes=268435456 count=0. Now grace=26843545
> doveadm(<user>): Debug: Namespace INBOX.: Using permissions from
> /home/.../test: mode=0700 gid=default
>
> $ cat maildirsize
> 268435456S
> 29394 34
>
>
> So the used quota has exactly been doubled by just creating an empty
> subfolder.
>
> Do you have any pointers for fixing my configuration or is this a bug in
> dovecot?
>
>
> I coincidentally resurrected a months-old thread with this same issue a
> few days ago. I'm seeing the exact same after upgrading from 2.2.32 to
> 2.2.36.
>
> The original poster (who also narrowed it down to something in 2.2.34)
> mentioned a workaround that does indeed work, namely setting
> mailbox_list_index=no:
>
> > doveadm -o 'mailbox_list_index=no' quota recalc -u myuser
>
> I've been staring at diffs of 2.2.33 and 2.2.34 without anything jumping
> out at me (not a C guy, sadly). Maybe src/lib-storage/index/index-storage.c
> or src/lib-storage/list/mailbox-list-fs-iter.c or
> src/lib-storage/list/mailbox-list-index-iter.c
> or src/lib-storage/list/mailbox-list-index.c?
>
> The latter few have some added strcmp's against "INBOX". Then again,
> there's a lot of new code in the diffs under src/lib-storage that
> references INBOX specifically.
>
>
> Can the Dovecot team confirm whether this is indeed a bug or not?  I've
> not yet been able to test 2.3.x to see if the problem exists there as well.
>
>
> I've bisected this down to this commit:
>
> git diff
> 7620195ceeea805137cbd1bae104e385eee474a9..97473a513feb2bbd763051869c8b7b83e24b37fa
>
>
> diff --git a/src/lib-storage/list/mailbox-list-index-iter.c
> b/src/lib-storage/list/mailbox-list-index-iter.c
> index c9afc7a..49cd941 100644
> --- a/src/lib-storage/list/mailbox-list-index-iter.c
> +++ b/src/lib-storage/list/mailbox-list-index-iter.c
> @@ -90,13 +90,18 @@ mailbox_list_index_update_info(struct
> mailbox_list_index_iterate_context *ctx)
>         if (strcmp(ctx->info.vname, "INBOX") != 0) {
>                 /* non-INBOX */
>                 ctx->info.vname = p_strdup(ctx->info_pool,
> ctx->info.vname);
> -       } else {
> +       } else if (!ctx->prefix_inbox_list) {
>                 /* listing INBOX itself */
>                 ctx->info.vname = "INBOX";
>                 if (mail_namespace_is_inbox_noinferiors(ctx->info.ns)) {
>                         ctx->info.flags &=
> ~(MAILBOX_CHILDREN|MAILBOX_NOCHILDREN);
>                         ctx->info.flags |= MAILBOX_NOINFERIORS;
>                 }
> +       } else {
> +               /* listing INBOX/INBOX */
> +               ctx->info.vname = p_strconcat(ctx->info_pool,
> +                       ctx->ctx.list->ns->prefix, "INBOX", NULL);
> +               ctx->info.flags |= MAILBOX_NONEXISTENT;
>         }
>         if ((node->flags & MAILBOX_LIST_INDEX_FLAG_NONEXISTENT) != 0)
>                 ctx->info.flags |= MAILBOX_NONEXISTENT;
> @@ -126,6 +131,14 @@ mailbox_list_index_update_next(struct
> mailbox_list_index_iterate_context *ctx,
>  {
>         struct mailbox_list_index_node *node = ctx->next_node;
>
> +       if (!ctx->prefix_inbox_list && ctx->ctx.list->ns->prefix_len > 0
> &&
> +           strcmp(node->name, "INBOX") == 0 && node->parent == NULL &&
> +           node->children != NULL) {
> +               /* prefix/INBOX has children */
> +               ctx->prefix_inbox_list = TRUE;
> +               return;
> +       }
> +
>         if (node->children != NULL && follow_children) {
>                 ctx->parent_len = str_len(ctx->path);
>                 ctx->next_node = node->children;
> diff --git a/src/lib-storage/list/mailbox-list-index.h
> b/src/lib-storage/list/mailbox-list-index.h
> index 9dc1c48..e4b042a 100644
> --- a/src/lib-storage/list/mailbox-list-index.h
> +++ b/src/lib-storage/list/mailbox-list-index.h
> @@ -138,6 +138,7 @@ struct mailbox_list_index_iterate_context {
>         struct mailbox_list_index_node *next_node;
>
>         unsigned int failed:1;
> +       unsigned int prefix_inbox_list:1;
>  };
>
>  extern MODULE_CONTEXT_DEFINE(mailbox_list_index_module,
>
>
> Prior to this commit, anything updating the quota would do the right thing
> for any .INBOX.<something> folders (i.e. not double count the contents of
> "INBOX" against the quota). After this commit, anything updating quota (new
> mail, quota recalc, etc) does the double counting of INBOX.
>
>
> Thank you for the bisect! We'll look into this.
>
> ---
> Aki Tuomi
>
>

Hi. I was curious if there were any fixes for this? We're still affected by
this (and I imagine others are too but don't realize it). Thanks!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://dovecot.org/pipermail/dovecot/attachments/20190806/caeebe0b/attachment-0001.html>