Inbox quota usage doubled when mailbox_list_index enabled, under some circumstances
Hello,
I searched through the list archives for anything that appeared to be similar to this but I didn't find any good matches. I apologize if this has been brought up before.
Beginning with Dovecot 2.2.34, reported quota usage of a user's inbox can be doubled when the following criteria are met:
- quota plugin is enabled
- mailbox_list_index=yes
- A sub-folder of the inbox exists whose name also matches the prefix (for example, "INBOX.Test"), even if that folder has no contents. (Apparently, some mail clients such as K-9 Mail on Android can create these folders automatically)
Downgrading to 2.2.33.2 allows this configuration to work normally. I looked through the changes in git between 2.2.33.2 and 2.2.34 and they appeared extensive, so unfortunately I wasn't able to pinpoint a specific change that caused this.
Reproduced on a CentOS 7 server with Dovecot 2.2.34 built from git sources according to https://wiki2.dovecot.org/CompilingSource#Compiling_Dovecot_From_Git using the following minimal configuration:
========== root@centos7 [~]# dovecot -n # 2.2.34 (874deae): /usr/local/etc/dovecot/dovecot.conf # OS: Linux 3.10.0-862.9.1.el7.x86_64 x86_64 CentOS Linux release 7.5.1804 (Core) # Hostname: centos7.local mail_location = maildir:~/Maildir mail_plugins = quota mailbox_list_index = yes namespace inbox { inbox = yes location = prefix = INBOX. separator = . } passdb { driver = pam } plugin { quota = maildir:Mailbox } userdb { driver = passwd }
With the system user "myuser" and a minimal Maildir setup containing 1 email in the inbox, and one sub-folder named "INBOX.Test":
========== root@centos7 [~]# ls -lAR /home/myuser/Maildir /home/myuser/Maildir: total 4 drwxr-xr-x. 2 myuser myuser 77 Jul 25 09:36 cur drwxr-xr-x. 5 myuser myuser 56 Jul 25 15:19 .INBOX.Test -rw-r--r--. 1 myuser myuser 14 Jul 25 15:19 maildirsize drwxr-xr-x. 2 myuser myuser 6 Jul 25 09:36 new drwxr-xr-x. 2 myuser myuser 6 Jul 25 09:36 tmp
/home/myuser/Maildir/cur: total 12416 -rw-r--r--. 1 myuser myuser 12712627 Jul 25 09:07 1532529376.M543965P58007.centos7.local,S=12712627,W=12877782:2,S
/home/myuser/Maildir/.INBOX.Test: total 0 drwxr-xr-x. 2 myuser myuser 6 Jul 25 09:36 cur -rw-r--r--. 1 myuser myuser 0 Jul 25 09:36 maildirfolder drwxr-xr-x. 2 myuser myuser 6 Jul 25 09:36 new drwxr-xr-x. 2 myuser myuser 6 Jul 25 09:36 tmp
/home/myuser/Maildir/.INBOX.Test/cur: total 0
/home/myuser/Maildir/.INBOX.Test/new: total 0
/home/myuser/Maildir/.INBOX.Test/tmp: total 0
/home/myuser/Maildir/new: total 0
/home/myuser/Maildir/tmp: total 0
Given this configuration, when "mailbox_list_index" is enabled, after quota recalculation, quota reports that there are 2 messages and that double the amount of storage is used:
========== root@centos7 [~]# doveadm quota recalc -u myuser; doveadm quota get -u myuser Quota name Type Value Limit % Mailbox STORAGE 24830 - 0 Mailbox MESSAGE 2 - 0
When "mailbox_list_index" is disabled and quota is recalculated, it will report the message and storage counts correctly:
========== root@centos7 [~]# doveadm -o 'mailbox_list_index=no' quota recalc -u myuser; doveadm quota get -u myuser Quota name Type Value Limit % Mailbox STORAGE 12415 - 0 Mailbox MESSAGE 1 - 0
I am hoping that someone with much more familiarity with the code path involved here could take a look at this issue and possibly submit a fix for it -- that would be greatly appreciated. Please let me know if you need any more details regarding the replication steps, though I hope that I didn't miss anything.
Thank you!
--
Chris Dillon Technical Analyst III Migrations Specialist cPanel, Inc.
I failed to mention previously that the behavior also exists in latest 2.2.36 built from git sources, and I believe also exists in 2.3.2. Though, I wasn't able to successfully build release-2.3.2 or master-2.3 branches from git in the minimal test environment due to an odd compilation error that I have not yet attempted to resolve, but I used an internal binary build of 2.3.2 which has a few (though probably not affecting this particular issue) patches applied to it.
--
Chris Dillon Technical Analyst III Migrations Specialist cPanel, Inc.
Register Now for cPanel Conference 2018 Oct 1 - 3, 2018, Houston, Texas https://conference.cpanel.com
I’ve had the opportunity to test the same configuration with a fresh build of the git master branch (2.4.devel) and the issue also occurs there. I see that "mailbox_list_index = yes" is now enabled by default. It can still be disabled via "mailbox_list_index = no" which allows the quota to be calculated correctly.
========== root@ubuntu1804:~# dovecot -n # 2.4.devel (44282aeeb): /usr/local/etc/dovecot/dovecot.conf # OS: Linux 4.15.0-30-generic x86_64 Ubuntu 18.04.1 LTS # Hostname: ubuntu1804 mail_location = maildir:~/Maildir mail_plugins = quota namespace inbox { inbox = yes location = prefix = INBOX. separator = . } passdb { driver = pam } plugin { quota = maildir:Mailbox } userdb { driver = passwd }
(To summarize from my previous message -- other than "mailbox_list_index = yes", second most important part of replication is that there is at least one email in the real inbox and at least one sub-folder named "INBOX" in maildir format)
root@ubuntu1804:~# ls -ld /home/myuser/Maildir/cur/1532529376.M543965P58007.centos7.local\,S\=12712627\,W\=12877782\:2\,S /home/myuser/Maildir/.INBOX.Test/ -rw-rw-r-- 1 myuser myuser 12712627 Aug 14 18:28 '/home/myuser/Maildir/cur/1532529376.M543965P58007.centos7.local,S=12712627,W=12877782:2,S' drwxrwxr-x 5 myuser myuser 87 Aug 14 18:56 /home/myuser/Maildir/.INBOX.Test/
(In the following example usage is doubled, there is only one email)
root@ubuntu1804:~# doveadm quota recalc -u myuser; doveadm quota get -u myuser Quota name Type Value Limit % Mailbox STORAGE 24830 - 0 Mailbox MESSAGE 2 - 0
(In the following example it works correctly with mailbox_list_index disabled)
root@ubuntu1804:~# doveadm -o 'mailbox_list_index=no' quota recalc -u myuser; doveadm quota get -u myuser Quota name Type Value Limit % Mailbox STORAGE 12415 - 0 Mailbox MESSAGE 1 - 0
Best Regards
On Tue, Aug 14, 2018 at 12:31 PM Chris Dillon chris.dillon@cpanel.net wrote:
I’ve had the opportunity to test the same configuration with a fresh build of the git master branch (2.4.devel) and the issue also occurs there. I see that "mailbox_list_index = yes" is now enabled by default. It can still be disabled via "mailbox_list_index = no" which allows the quota to be calculated correctly.
========== root@ubuntu1804:~# dovecot -n # 2.4.devel (44282aeeb): /usr/local/etc/dovecot/dovecot.conf # OS: Linux 4.15.0-30-generic x86_64 Ubuntu 18.04.1 LTS # Hostname: ubuntu1804 mail_location = maildir:~/Maildir mail_plugins = quota namespace inbox { inbox = yes location = prefix = INBOX. separator = . } passdb { driver = pam } plugin { quota = maildir:Mailbox } userdb { driver = passwd }
(To summarize from my previous message -- other than "mailbox_list_index = yes", second most important part of replication is that there is at least one email in the real inbox and at least one sub-folder named "INBOX" in maildir format)
root@ubuntu1804:~# ls -ld /home/myuser/Maildir/cur/1532529376.M543965P58007.centos7.local\,S\=12712627\,W\=12877782\:2\,S /home/myuser/Maildir/.INBOX.Test/ -rw-rw-r-- 1 myuser myuser 12712627 Aug 14 18:28 '/home/myuser/Maildir/cur/1532529376.M543965P58007.centos7.local,S=12712627,W=12877782:2,S' drwxrwxr-x 5 myuser myuser 87 Aug 14 18:56 /home/myuser/Maildir/.INBOX.Test/
(In the following example usage is doubled, there is only one email)
root@ubuntu1804:~# doveadm quota recalc -u myuser; doveadm quota get -u myuser Quota name Type Value Limit % Mailbox STORAGE 24830 - 0 Mailbox MESSAGE 2 - 0
(In the following example it works correctly with mailbox_list_index disabled)
root@ubuntu1804:~# doveadm -o 'mailbox_list_index=no' quota recalc -u myuser; doveadm quota get -u myuser Quota name Type Value Limit % Mailbox STORAGE 12415 - 0 Mailbox MESSAGE 1 - 0
Best Regards
We recently upgraded from 2.2.32 to 2.2.36.1 and ran into the same issue as above. I was about to start compiling all the intermediate versions to pinpoint where it started and happened upon this post first. We are in the same boat where some users have *sub*folders starting with INBOX, resulting in directory names like ".INBOX.Event" (and our namespace root is INBOX) and quota calculation double-counts INBOX's vsize.
Just to be clear too: At least in my case, it's *not* double counting, e.g., INBOX.Event. But if the above conditions are met, it's double-counting *INBOX*, because it now sees a folder called INBOX.INBOX (which does *not* exist on the filesystem).
I hadn't gotten as far as Chris did (this just bubbled up today), but his solution works here too, i.e. passing -o 'mailbox_list_index=no' to doveadm quota recalc.
Also, if I rename the directories to something else (e.g. from above, rename ".INBOX.Event" to ".notINBOX.Event"), a quota recalc works just fine. The presence of a directory called .INBOX.<something> is triggering this.
I'm able to create new subfolders called INBOX.<something> with clients like Apple Mail, which was a bit surprising (Apple Mail however choked when I tried to just create a subfolder called 'INBOX'). We've got millions of mailboxes, so educating users is a non-starter :)
Any fix for this from the dovecot devs?
participants (2)
-
Chris Dillon
-
Mark Moseley