On Wed, Apr 3, 2019 at 9:37 PM Mark Moseley <moseleymark@gmail.com> wrote:

On Wed, Mar 20, 2019 at 2:13 PM Mark Moseley <moseleymark@gmail.com> wrote:

...

Just hoping to get some dev eyes on this. I'm incredibly reluctant to throw the word 'bug' around (since 99 times out of 100, it's not -- it's almost always the config), but I can't think of any way that this could be a config issue, esp when the pre-2.2.34 version works as expected.

I noticed during troubleshooting that dovecot errors out if I try to create a subfolder called 'INBOX' but it'll happily create a subfolder called INBOX.SomethingElse (i.e. a folder called INBOX.INBOX.SomethingElse - resulting in a directory called .INBOX.SomethingElse on the filesystem, and leading to the problem described below). Is that sub-subfolder creation (where the top level subfolder matches the namespace name) supposed to be allowed? It seems odd that 'INBOX' (as a subfolder of INBOX) would be blocked but INBOX.SomethingElse (as a subfolder of INBOX) would be allowed. I'd expect INBOX.SomethingElse (i.e. INBOX.INBOX.SomethingElse) would be blocked as well.

On Wed, Mar 13, 2019 at 4:46 AM Bernd Wurst via dovecot < dovecot@dovecot.org> wrote:

...

Hello,

we're operating dovecot on a small server. Some years ago, we migrated from courier IMAP to dovecot. Therefore, we defined our default Namespace "inbox" with prefix "INBOX." to have this compatible. I found this in some migration docs those days. Generally, everything worked as expected.

Our only namespace is configured like this:

namespace inbox { separator = . prefix = INBOX. inbox = yes }

Regularly, there is no folder named INBOX or .INBOX in the file system, I suppose this is correct. But I found a special corner case today when it comes to quota calculation.

When - for whatever reason - a folder .INBOX.foo (for arbitrary values of foo) exists, the whole mailbox is counted twice in quota recalculation. Just creating .INBOX does nothing but a subfolder triggers the problem.

This is my shell view (replaced username and file path and deleted unnecessary debug output)

$ cat maildirsize 268435456S 14697 17 $ maildirmake .INBOX.foo $ sudo doveadm -D quota recalc -u <user> [...] doveadm(<user>): Debug: Namespace inbox: type=private, prefix=INBOX., sep=., inbox=yes, hidden=no, list=yes, subscriptions=yes location=maildir:/home/.../test doveadm(<user>): Debug: maildir++: root=/home/.../test, index=, indexpvt=, control=, inbox=/home/.../test, alt= doveadm(<user>): Debug: Namespace : type=private, prefix=, sep=, inbox=no, hidden=yes, list=no, subscriptions=no location=fail::LAYOUT=none doveadm(<user>): Debug: none: root=, index=, indexpvt=, control=, inbox=, alt= doveadm(<user>): Debug: quota: quota_over_flag check: quota_over_script unset - skipping doveadm(<user>): Debug: Quota root User quota: Recalculated relative rules with bytes=268435456 count=0. Now grace=26843545 doveadm(<user>): Debug: Namespace INBOX.: Using permissions from /home/.../test: mode=0700 gid=default

$ cat maildirsize 268435456S 29394 34

So the used quota has exactly been doubled by just creating an empty subfolder.

Do you have any pointers for fixing my configuration or is this a bug in dovecot?

I coincidentally resurrected a months-old thread with this same issue a few days ago. I'm seeing the exact same after upgrading from 2.2.32 to 2.2.36.

The original poster (who also narrowed it down to something in 2.2.34) mentioned a workaround that does indeed work, namely setting mailbox_list_index=no:

...

doveadm -o 'mailbox_list_index=no' quota recalc -u myuser

I've been staring at diffs of 2.2.33 and 2.2.34 without anything jumping out at me (not a C guy, sadly). Maybe src/lib-storage/index/index-storage.c or src/lib-storage/list/mailbox-list-fs-iter.c or src/lib-storage/list/mailbox-list-index-iter.c or src/lib-storage/list/mailbox-list-index.c?

The latter few have some added strcmp's against "INBOX". Then again, there's a lot of new code in the diffs under src/lib-storage that references INBOX specifically.

Can the Dovecot team confirm whether this is indeed a bug or not? I've not yet been able to test 2.3.x to see if the problem exists there as well.

I've bisected this down to this commit: git diff 7620195ceeea805137cbd1bae104e385eee474a9..97473a513feb2bbd763051869c8b7b83e24b37fa diff --git a/src/lib-storage/list/mailbox-list-index-iter.c b/src/lib-storage/list/mailbox-list-index-iter.c index c9afc7a..49cd941 100644 --- a/src/lib-storage/list/mailbox-list-index-iter.c +++ b/src/lib-storage/list/mailbox-list-index-iter.c @@ -90,13 +90,18 @@ mailbox_list_index_update_info(struct mailbox_list_index_iterate_context *ctx) if (strcmp(ctx->info.vname, "INBOX") != 0) { /* non-INBOX */ ctx->info.vname = p_strdup(ctx->info_pool, ctx->info.vname); - } else { + } else if (!ctx->prefix_inbox_list) { /* listing INBOX itself */ ctx->info.vname = "INBOX"; if (mail_namespace_is_inbox_noinferiors(ctx->info.ns)) { ctx->info.flags &= ~(MAILBOX_CHILDREN|MAILBOX_NOCHILDREN); ctx->info.flags |= MAILBOX_NOINFERIORS; } + } else { + /* listing INBOX/INBOX */ + ctx->info.vname = p_strconcat(ctx->info_pool, + ctx->ctx.list->ns->prefix, "INBOX", NULL); + ctx->info.flags |= MAILBOX_NONEXISTENT; } if ((node->flags & MAILBOX_LIST_INDEX_FLAG_NONEXISTENT) != 0) ctx->info.flags |= MAILBOX_NONEXISTENT; @@ -126,6 +131,14 @@ mailbox_list_index_update_next(struct mailbox_list_index_iterate_context *ctx, { struct mailbox_list_index_node *node = ctx->next_node; + if (!ctx->prefix_inbox_list && ctx->ctx.list->ns->prefix_len > 0 && + strcmp(node->name, "INBOX") == 0 && node->parent == NULL && + node->children != NULL) { + /* prefix/INBOX has children */ + ctx->prefix_inbox_list = TRUE; + return; + } + if (node->children != NULL && follow_children) { ctx->parent_len = str_len(ctx->path); ctx->next_node = node->children; diff --git a/src/lib-storage/list/mailbox-list-index.h b/src/lib-storage/list/mailbox-list-index.h index 9dc1c48..e4b042a 100644 --- a/src/lib-storage/list/mailbox-list-index.h +++ b/src/lib-storage/list/mailbox-list-index.h @@ -138,6 +138,7 @@ struct mailbox_list_index_iterate_context { struct mailbox_list_index_node *next_node; unsigned int failed:1; + unsigned int prefix_inbox_list:1; }; extern MODULE_CONTEXT_DEFINE(mailbox_list_index_module, Prior to this commit, anything updating the quota would do the right thing for any .INBOX.<something> folders (i.e. not double count the contents of "INBOX" against the quota). After this commit, anything updating quota (new mail, quota recalc, etc) does the double counting of INBOX.

On Tue, Apr 9, 2019 at 9:52 PM Aki Tuomi <aki.tuomi@open-xchange.com> wrote:

On 10 April 2019 05:00 Mark Moseley via dovecot <dovecot@dovecot.org> wrote:

On Wed, Apr 3, 2019 at 9:37 PM Mark Moseley < moseleymark@gmail.com> wrote:

On Wed, Mar 20, 2019 at 2:13 PM Mark Moseley < moseleymark@gmail.com> wrote:

Just hoping to get some dev eyes on this. I'm incredibly reluctant to throw the word 'bug' around (since 99 times out of 100, it's not -- it's almost always the config), but I can't think of any way that this could be a config issue, esp when the pre-2.2.34 version works as expected.

I noticed during troubleshooting that dovecot errors out if I try to create a subfolder called 'INBOX' but it'll happily create a subfolder called INBOX.SomethingElse (i.e. a folder called INBOX.INBOX.SomethingElse - resulting in a directory called .INBOX.SomethingElse on the filesystem, and leading to the problem described below). Is that sub-subfolder creation (where the top level subfolder matches the namespace name) supposed to be allowed? It seems odd that 'INBOX' (as a subfolder of INBOX) would be blocked but INBOX.SomethingElse (as a subfolder of INBOX) would be allowed. I'd expect INBOX.SomethingElse (i.e. INBOX.INBOX.SomethingElse) would be blocked as well.

On Wed, Mar 13, 2019 at 4:46 AM Bernd Wurst via dovecot < dovecot@dovecot.org> wrote:

Hello,

we're operating dovecot on a small server. Some years ago, we migrated from courier IMAP to dovecot. Therefore, we defined our default Namespace "inbox" with prefix "INBOX." to have this compatible. I found this in some migration docs those days. Generally, everything worked as expected.

Our only namespace is configured like this:

namespace inbox { separator = . prefix = INBOX. inbox = yes }

Regularly, there is no folder named INBOX or .INBOX in the file system, I suppose this is correct. But I found a special corner case today when it comes to quota calculation.

When - for whatever reason - a folder .INBOX.foo (for arbitrary values of foo) exists, the whole mailbox is counted twice in quota recalculation. Just creating .INBOX does nothing but a subfolder triggers the problem.

This is my shell view (replaced username and file path and deleted unnecessary debug output)

$ cat maildirsize 268435456S 14697 17 $ maildirmake .INBOX.foo $ sudo doveadm -D quota recalc -u <user> [...] doveadm(<user>): Debug: Namespace inbox: type=private, prefix=INBOX., sep=., inbox=yes, hidden=no, list=yes, subscriptions=yes location=maildir:/home/.../test doveadm(<user>): Debug: maildir++: root=/home/.../test, index=, indexpvt=, control=, inbox=/home/.../test, alt= doveadm(<user>): Debug: Namespace : type=private, prefix=, sep=, inbox=no, hidden=yes, list=no, subscriptions=no location=fail::LAYOUT=none doveadm(<user>): Debug: none: root=, index=, indexpvt=, control=, inbox=, alt= doveadm(<user>): Debug: quota: quota_over_flag check: quota_over_script unset - skipping doveadm(<user>): Debug: Quota root User quota: Recalculated relative rules with bytes=268435456 count=0. Now grace=26843545 doveadm(<user>): Debug: Namespace INBOX.: Using permissions from /home/.../test: mode=0700 gid=default

$ cat maildirsize 268435456S 29394 34

So the used quota has exactly been doubled by just creating an empty subfolder.

Do you have any pointers for fixing my configuration or is this a bug in dovecot?

I coincidentally resurrected a months-old thread with this same issue a few days ago. I'm seeing the exact same after upgrading from 2.2.32 to 2.2.36.

The original poster (who also narrowed it down to something in 2.2.34) mentioned a workaround that does indeed work, namely setting mailbox_list_index=no:

...

doveadm -o 'mailbox_list_index=no' quota recalc -u myuser

I've been staring at diffs of 2.2.33 and 2.2.34 without anything jumping out at me (not a C guy, sadly). Maybe src/lib-storage/index/index-storage.c or src/lib-storage/list/mailbox-list-fs-iter.c or src/lib-storage/list/mailbox-list-index-iter.c or src/lib-storage/list/mailbox-list-index.c?

The latter few have some added strcmp's against "INBOX". Then again, there's a lot of new code in the diffs under src/lib-storage that references INBOX specifically.

Can the Dovecot team confirm whether this is indeed a bug or not? I've not yet been able to test 2.3.x to see if the problem exists there as well.

I've bisected this down to this commit:

git diff 7620195ceeea805137cbd1bae104e385eee474a9..97473a513feb2bbd763051869c8b7b83e24b37fa

diff --git a/src/lib-storage/list/mailbox-list-index-iter.c b/src/lib-storage/list/mailbox-list-index-iter.c index c9afc7a..49cd941 100644 --- a/src/lib-storage/list/mailbox-list-index-iter.c +++ b/src/lib-storage/list/mailbox-list-index-iter.c @@ -90,13 +90,18 @@ mailbox_list_index_update_info(struct mailbox_list_index_iterate_context *ctx) if (strcmp(ctx->info.vname, "INBOX") != 0) { /* non-INBOX */ ctx->info.vname = p_strdup(ctx->info_pool, ctx->info.vname); - } else { + } else if (!ctx->prefix_inbox_list) { /* listing INBOX itself */ ctx->info.vname = "INBOX"; if (mail_namespace_is_inbox_noinferiors(ctx->info.ns)) { ctx->info.flags &= ~(MAILBOX_CHILDREN|MAILBOX_NOCHILDREN); ctx->info.flags |= MAILBOX_NOINFERIORS; } + } else { + /* listing INBOX/INBOX */ + ctx->info.vname = p_strconcat(ctx->info_pool, + ctx->ctx.list->ns->prefix, "INBOX", NULL); + ctx->info.flags |= MAILBOX_NONEXISTENT; } if ((node->flags & MAILBOX_LIST_INDEX_FLAG_NONEXISTENT) != 0) ctx->info.flags |= MAILBOX_NONEXISTENT; @@ -126,6 +131,14 @@ mailbox_list_index_update_next(struct mailbox_list_index_iterate_context *ctx, { struct mailbox_list_index_node *node = ctx->next_node;

+ if (!ctx->prefix_inbox_list && ctx->ctx.list->ns->prefix_len > 0 && + strcmp(node->name, "INBOX") == 0 && node->parent == NULL && + node->children != NULL) { + /* prefix/INBOX has children */ + ctx->prefix_inbox_list = TRUE; + return; + } + if (node->children != NULL && follow_children) { ctx->parent_len = str_len(ctx->path); ctx->next_node = node->children; diff --git a/src/lib-storage/list/mailbox-list-index.h b/src/lib-storage/list/mailbox-list-index.h index 9dc1c48..e4b042a 100644 --- a/src/lib-storage/list/mailbox-list-index.h +++ b/src/lib-storage/list/mailbox-list-index.h @@ -138,6 +138,7 @@ struct mailbox_list_index_iterate_context { struct mailbox_list_index_node *next_node;

unsigned int failed:1; + unsigned int prefix_inbox_list:1; };

extern MODULE_CONTEXT_DEFINE(mailbox_list_index_module,

Prior to this commit, anything updating the quota would do the right thing for any .INBOX.<something> folders (i.e. not double count the contents of "INBOX" against the quota). After this commit, anything updating quota (new mail, quota recalc, etc) does the double counting of INBOX.

Thank you for the bisect! We'll look into this.

--- Aki Tuomi

Hi. I was curious if there were any fixes for this? We're still affected by this (and I imagine others are too but don't realize it). Thanks!