[Dovecot] Wrong quota calculation
Hi, We've a strange quota behavior on our dovecot servers. It seems that it's somehow doubling the real used storage (or could be something like twice the inbox size + size of other folders, hard to know for sure). It was initial setup with the maildir++ system. Now it's been configured with dict and mysql table but the result is the same.
Dovecot version : 2.1.7 (from squeeze-backports). Maildir format is used As example, let's use mailbox@domain.com. If I check in the quota table, I'll find :
- size : 697151354
- number of files : 1510
Now if I check the Maildir folder on the filesystem, I'll find :
- size : 347960000
- number of files : 824 (using find . -type f|grep -v dovecot|wc -l, not 100% accurate)
Another mail box Reported by dovecot : 503446986 1081 Filesystem : 248592000 598
Does anyone has an idea of what could be possibly wrong. We've checked & double-checked everything and nothing seems to be wrong (no dsync used).
-- Regards Sébastien
Cibest ML wrote:
We've a strange quota behavior on our dovecot servers. It seems that it's somehow doubling the real used storage (or could be something like twice the inbox size + size of other folders, hard to know for sure). It was initial setup with the maildir++ system. Now it's been configured with dict and mysql table but the result is the same.
Dovecot version : 2.1.7 (from squeeze-backports). Maildir format is used As example, let's use mailbox@domain.com. If I check in the quota table, I'll find :
- size : 697151354
- number of files : 1510
Now if I check the Maildir folder on the filesystem, I'll find :
- size : 347960000
- number of files : 824 (using find . -type f|grep -v dovecot|wc -l, not 100% accurate)
Another mail box Reported by dovecot : 503446986 1081 Filesystem : 248592000 598
Does anyone has an idea of what could be possibly wrong. We've checked & double-checked everything and nothing seems to be wrong (no dsync used).
What happens after quota resync? Does the quota change?
What happens to the "doveadm quota get" output after LMTP/LDA delivery of a mail with a fixed-size, let's say one mail with 1 MB?
How does your setup look like? Mailbox+Director Proxy or only Mailboxes?
What is your doveconf -n output of mailbox (and possibly director)?
Regards Daniel
On 08/09/2012 01:38, Daniel Parthey wrote:
Cibest ML wrote:
We've a strange quota behavior on our dovecot servers. It seems that it's somehow doubling the real used storage (or could be something like twice the inbox size + size of other folders, hard to know for sure). It was initial setup with the maildir++ system. Now it's been configured with dict and mysql table but the result is the same.
Dovecot version : 2.1.7 (from squeeze-backports). Maildir format is used As example, let's use mailbox@domain.com. If I check in the quota table, I'll find :
- size : 697151354
- number of files : 1510
Now if I check the Maildir folder on the filesystem, I'll find :
- size : 347960000
- number of files : 824 (using find . -type f|grep -v dovecot|wc -l, not 100% accurate)
Another mail box Reported by dovecot : 503446986 1081 Filesystem : 248592000 598
Does anyone has an idea of what could be possibly wrong. We've checked & double-checked everything and nothing seems to be wrong (no dsync used). What happens after quota resync? Does the quota change?
What happens to the "doveadm quota get" output after LMTP/LDA delivery of a mail with a fixed-size, let's say one mail with 1 MB?
How does your setup look like? Mailbox+Director Proxy or only Mailboxes?
What is your doveconf -n output of mailbox (and possibly director)?
Regards Daniel Hi Daniel, Thanks for your answer Here are some tests to show the problem. The test mailbox is initialy empty. All described actions are done in a sequencial way (test mail with an attachement of ~2Mo)
Message delivered by LMTP Quota name Type Value Limit User quota STORAGE 2653 524288 User quota MESSAGE 1 -
quota recalc Quota name Type Value Limit User quota STORAGE 5307 524288 User quota MESSAGE 2 -
move to draft + compact Quota name Type Value Limit User quota STORAGE 2653 524288 User quota MESSAGE 1 -
recalc Quota name Type Value Limit User quota STORAGE 2653 524288 User quota MESSAGE 1 -
move to subfolder "test" in inbox User quota STORAGE 2653 524288 User quota MESSAGE 1 -
move message back to inbox Quota name Type Value Limit User quota STORAGE 5306 524288 User quota MESSAGE 2 -
compact Quota name Type Value Limit User quota STORAGE 2653 524288 User quota MESSAGE 1 -
recalc quota Quota name Type Value Limit User quota STORAGE 5306 524288 User quota MESSAGE 2 -
No dovecot director is used at the moment (it's planned). There's one server handling incoming mails (postfix + LMTP), one for IMAP/POP (dovecot) & one for email sending (postfix).
dovecot -n output of incoming mails server
# 2.1.7: /etc/dovecot/dovecot.conf # OS: Linux 3.4.1 Debian 6.0.5 ext4 dict { quota = mysql:/etc/dovecot/dovecot-dict-sql.conf.ext } mail_gid = 1500 mail_home = /data/hosting/mail%h/%d/%n mail_location = maildir:/data/hosting/mail%h/%d/%n/Maildir mail_nfs_storage = yes mail_plugins = " quota autocreate" mail_uid = 1500 managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave namespace inbox { inbox = yes location = prefix = } passdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql } plugin { autocreate = Inbox autocreate2 = Trash autocreate3 = Sent autocreate4 = Spam autocreate5 = Drafts autosubscribe = Inbox autosubscribe2 = Trash autosubscribe3 = Sent autosubscribe4 = Spam autosubscribe5 = Drafts quota = dict:User quota::noenforcing:proxy::quota quota_rule = *:storage=512M quota_rule2 = Trash:storage=+100M quota_warning = storage=90%% quota-warning 95 %u quota_warning2 = storage=80%% quota-warning 80 %u sieve = /data/hosting/mail%h/%d/%n/sieve/.dovecot.sieve sieve_before = /etc/dovecot/globalsieve sieve_dir = /data/hosting/mail%h/%d/%n/sieve/ } protocols = " lmtp sieve" quota_full_tempfail = yes service auth { unix_listener auth-userdb { group = vmail mode = 0660 user = vmail } } service dict { unix_listener dict { group = vmail mode = 0600 user = vmail } } service lmtp { inet_listener lmtp { address = 10.3.65.71 127.0.0.1 ::1 port = 2055 } unix_listener /var/spool/postfix/private/dovecot-lmtp { group = postfix mode = 0660 user = postfix } user = vmail } ssl_cert = </etc/ssl/certs/dovecot.pem ssl_key = </etc/ssl/private/dovecot.pem userdb { driver = prefetch } userdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql } protocol lmtp { mail_plugins = " quota sieve autocreate" } protocol lda { mail_plugins = " quota" }
*IMAP/POP server conf*
# 2.1.7: /etc/dovecot/dovecot.conf # OS: Linux 3.4.1-ipv6-64-cibest-grsec x86_64 Debian 6.0.5 ext4 auth_mechanisms = plain login dict { quota = mysql:/etc/dovecot/dovecot-dict-sql.conf.ext } disable_plaintext_auth = no mail_gid = 1500 mail_home = /data/hosting/mail%h/%d/%n mail_location = maildir:/data/hosting/mail%h/%d/%n/Maildir:INDEX=/data/hosting/mailcache%h/%d/%n mail_nfs_storage = yes mail_plugins = " quota autocreate" mail_uid = 1500 managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave namespace inbox { inbox = yes location = prefix = } passdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql } plugin { autocreate = Inbox autocreate2 = Trash autocreate3 = Sent autocreate4 = Spam autocreate5 = Drafts autosubscribe = Inbox autosubscribe2 = Trash autosubscribe3 = Sent autosubscribe4 = Spam autosubscribe5 = Drafts quota = dict:User quota::noenforcing:proxy::quota quota_rule = *:storage=512M quota_rule2 = Trash:storage=+100M sieve = /data/hosting/mail%h/%d/%n/sieve/.dovecot.sieve sieve_before = sieve_before = /etc/dovecot/globalsieve sieve_dir = /data/hosting/mail%h/%d/%n/sieve/ } protocols = " imap sieve pop3" service auth-worker { user = $default_internal_user } service auth { inet_listener { port = 63200 } unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0666 user = postfix } unix_listener auth-userdb { group = vmail mode = 0660 user = vmail } } service dict { unix_listener dict { group = vmail mode = 0600 user = vmail } } service imap-login { inet_listener imap { port = 143 } inet_listener imaps { port = 993 ssl = yes } process_min_avail = 4 service_count = 1 vsz_limit = 256 M } service managesieve-login { inet_listener sieve { port = 4190 } process_min_avail = 5 service_count = 0 } service pop3-login { inet_listener pop3 { port = 110 } inet_listener pop3s { port = 995 ssl = yes } } ssl_ca = /etc/ssl/certs/xxxSSLCA.pem ssl_cert = </etc/ssl/certs/xxx_com.crt ssl_key = </etc/ssl/private/xxx_com.key userdb { driver = prefetch } protocol lda { mail_plugins = " quota" } protocol imap { mail_plugins = " quota imap_quota autocreate" } protocol pop3 { mail_plugins = " quota" }
Hope it'll help nailing the issue
-- Regards Sébastien
Cibest ML wrote:
On 08/09/2012 01:38, Daniel Parthey wrote:
Cibest ML wrote:
We've a strange quota behavior on our dovecot servers. It seems that it's somehow doubling the real used storage (or could be something like twice the inbox size + size of other folders, hard to know for sure). It was initial setup with the maildir++ system. Now it's been configured with dict and mysql table but the result is the same.
Here are some tests to show the problem. The test mailbox is initialy empty. All described actions are done in a sequencial way (test mail with an attachement of ~2MB)
- Message delivered by LMTP Quota name Type Value Limit User quota STORAGE 2653 524288 User quota MESSAGE 1 -
Seems ok.
- quota recalc Quota name Type Value Limit User quota STORAGE 5307 524288 User quota MESSAGE 2 -
Strange. There must be some issue with quota recalculation. Maybe there are two instances of the message in the maildir? How large is the actual maildir measured by diskusage (du) command and where are the messages stored in the filesystem?
- move to draft + compact Quota name Type Value Limit User quota STORAGE 2653 524288 User quota MESSAGE 1 -
Seems absolutely correct. Message is copied, deleted and removed from the old directory.
- recalc Quota name Type Value Limit User quota STORAGE 2653 524288 User quota MESSAGE 1 -
Seems after compacting the folders there is only one instance of the message.
- move to subfolder "test" in inbox User quota STORAGE 2653 524288 User quota MESSAGE 1 -
- move message back to inbox Quota name Type Value Limit User quota STORAGE 5306 524288 User quota MESSAGE 2 -
The message gets copied and deleted, so there are 2 messages.
- compact Quota name Type Value Limit User quota STORAGE 2653 524288 User quota MESSAGE 1 -
After compaction the "copied+deleted" message gets finally removed.
- recalc quota Quota name Type Value Limit User quota STORAGE 5306 524288 User quota MESSAGE 2 -
Very strange. The removed message "reappeared"? Are you using some network filesystem and possibly experience filesystem caching effects? In this case a dovecot director would help.
Regards Daniel
On 09/09/2012 23:30, Daniel Parthey wrote:
Cibest ML wrote:
On 08/09/2012 01:38, Daniel Parthey wrote:
Cibest ML wrote:
We've a strange quota behavior on our dovecot servers. It seems that it's somehow doubling the real used storage (or could be something like twice the inbox size + size of other folders, hard to know for sure). It was initial setup with the maildir++ system. Now it's been configured with dict and mysql table but the result is the same. Here are some tests to show the problem. The test mailbox is initialy empty. All described actions are done in a sequencial way (test mail with an attachement of ~2MB)
- Message delivered by LMTP Quota name Type Value Limit User quota STORAGE 2653 524288 User quota MESSAGE 1 - Seems ok.
- quota recalc Quota name Type Value Limit User quota STORAGE 5307 524288 User quota MESSAGE 2 - Strange. There must be some issue with quota recalculation. Maybe there are two instances of the message in the maildir? How large is the actual maildir measured by diskusage (du) command and where are the messages stored in the filesystem?
- move to draft + compact Quota name Type Value Limit User quota STORAGE 2653 524288 User quota MESSAGE 1 - Seems absolutely correct. Message is copied, deleted and removed from the old directory.
- recalc Quota name Type Value Limit User quota STORAGE 2653 524288 User quota MESSAGE 1 - Seems after compacting the folders there is only one instance of the message.
- move to subfolder "test" in inbox User quota STORAGE 2653 524288 User quota MESSAGE 1 -
- move message back to inbox Quota name Type Value Limit User quota STORAGE 5306 524288 User quota MESSAGE 2 - The message gets copied and deleted, so there are 2 messages.
- compact Quota name Type Value Limit User quota STORAGE 2653 524288 User quota MESSAGE 1 - After compaction the "copied+deleted" message gets finally removed.
- recalc quota Quota name Type Value Limit User quota STORAGE 5306 524288 User quota MESSAGE 2 - Very strange. The removed message "reappeared"? Are you using some network filesystem and possibly experience filesystem caching effects? In this case a dovecot director would help.
Regards Daniel
The storage is indeed done via NFS. There might be some caching effects but in that case, the behaviour should occur for all the test cases. But it seems only to happen if the mails are stored in INBOX, not in a subfolder of INBOX nor another IMAP folder. We've never been able to reproduce the issue if the email is not exactly in INBOX. We've added some options to check if it's a NFS issue (but it shouldn't be needed as indexes aren't on a NFS filer) : map_disable = yes mail_fsync = always mail_nfs_index = yes It doesn't change anything, same behavior.
Here are some test cases to show that there's always 1 mail stored on the filesystem
new incoming mail
du -> 8 ls -> -rw------- 1 vmail vmail 3748 Sep 10 16:11 1347286291.M194531P4329.mspooll01,S=3748,W=3847
quota get
Quota name Type Value Limit User quota STORAGE 3 524288 User quota MESSAGE 1 -
du -> 8 ls -> -rw------- 1 vmail vmail 3748 Sep 10 16:11 1347286291.M194531P4329.mspooll01,S=3748,W=3847
quota recacl
Quota name Type Value Limit User quota STORAGE 7 524288 User quota MESSAGE 2 -
du -> 8 ls -> -rw------- 1 vmail vmail 3748 Sep 10 16:11 1347286291.M194531P4329.mspooll01,S=3748,W=3847
it really seems to be connected to the recalc operation.
Moving to dovecot director would require some work but at the moment, we can't be sure it'd solve the issue (as we only have one IMAP/POP server
- one LMTP server, we shouldn't have issues with stickiness of connexions)
-- Regards Sébastien
On 8.9.2012, at 18.07, Cibest ML wrote:
Message delivered by LMTP Quota name Type Value Limit User quota STORAGE 2653 524288 User quota MESSAGE 1 -
quota recalc Quota name Type Value Limit User quota STORAGE 5307 524288 User quota MESSAGE 2 -
Quota recalculation is clearly seeing the message twice. My first guess was that you had two namespaces pointing to the same location, but you seem to have only one. Do you have symlinks in your Maildir?
On 11/09/2012 20:48, Timo Sirainen wrote:
On 8.9.2012, at 18.07, Cibest ML wrote:
Message delivered by LMTP Quota name Type Value Limit User quota STORAGE 2653 524288 User quota MESSAGE 1 -
quota recalc Quota name Type Value Limit User quota STORAGE 5307 524288 User quota MESSAGE 2 - Quota recalculation is clearly seeing the message twice. My first guess was that you had two namespaces pointing to the same location, but you seem to have only one. Do you have symlinks in your Maildir?
Hi, Nothing specific on the namespace side. The only entry is the default one in 10-mail.conf namespace inbox { inbox = yes }
I tried to comment it out but it doesn't change anything
And no symlinks in the Maildir. Here's the file structure of Maildir for the test mailbox
./dovecot-uidvalidity.504df4e7 ./dovecot-uidvalidity ./.Sent ./.Sent/maildirfolder ./.Sent/new ./.Sent/dovecot.index.log ./.Sent/tmp ./.Sent/dovecot-uidlist ./.Sent/cur ./.Spam ./.Spam/maildirfolder ./.Spam/new ./.Spam/dovecot.index.log ./.Spam/tmp ./.Spam/dovecot-uidlist ./.Spam/cur ./new ./subscriptions ./.Drafts ./.Drafts/maildirfolder ./.Drafts/new ./.Drafts/dovecot.index.log ./.Drafts/tmp ./.Drafts/dovecot-uidlist ./.Drafts/cur ./.Drafts/dovecot-keywords ./dovecot.index.log ./tmp ./dovecot.mailbox.log ./dovecot-uidlist ./.Trash ./.Trash/maildirfolder ./.Trash/new ./.Trash/dovecot.index.log ./.Trash/tmp ./.Trash/dovecot-uidlist ./.Trash/cur ./cur ./cur/1347286291.M194531P4329.mspooll01,S=3748,W=3847:2,Sa ./dovecot-keywords ./dovecot.index.cache
-- Regards Sébastien
On 12/09/2012 10:58, Cibest ML wrote:
On 11/09/2012 20:48, Timo Sirainen wrote:
On 8.9.2012, at 18.07, Cibest ML wrote:
Message delivered by LMTP Quota name Type Value Limit User quota STORAGE 2653 524288 User quota MESSAGE 1 -
quota recalc Quota name Type Value Limit User quota STORAGE 5307 524288 User quota MESSAGE 2 - Quota recalculation is clearly seeing the message twice. My first guess was that you had two namespaces pointing to the same location, but you seem to have only one. Do you have symlinks in your Maildir?
Hi, Nothing specific on the namespace side. The only entry is the default one in 10-mail.conf namespace inbox { inbox = yes }
I tried to comment it out but it doesn't change anything
And no symlinks in the Maildir. Here's the file structure of Maildir for the test mailbox
./dovecot-uidvalidity.504df4e7 ./dovecot-uidvalidity ./.Sent ./.Sent/maildirfolder ./.Sent/new ./.Sent/dovecot.index.log ./.Sent/tmp ./.Sent/dovecot-uidlist ./.Sent/cur ./.Spam ./.Spam/maildirfolder ./.Spam/new ./.Spam/dovecot.index.log ./.Spam/tmp ./.Spam/dovecot-uidlist ./.Spam/cur ./new ./subscriptions ./.Drafts ./.Drafts/maildirfolder ./.Drafts/new ./.Drafts/dovecot.index.log ./.Drafts/tmp ./.Drafts/dovecot-uidlist ./.Drafts/cur ./.Drafts/dovecot-keywords ./dovecot.index.log ./tmp ./dovecot.mailbox.log ./dovecot-uidlist ./.Trash ./.Trash/maildirfolder ./.Trash/new ./.Trash/dovecot.index.log ./.Trash/tmp ./.Trash/dovecot-uidlist ./.Trash/cur ./cur ./cur/1347286291.M194531P4329.mspooll01,S=3748,W=3847:2,Sa ./dovecot-keywords ./dovecot.index.cache
-- Regards Sébastien
We did some more investigations as we had to enable the no enforcing quota option to prevent too many compltains from our customers. As the issue seems to be linked to the inbox folder, we tried to focus on that. According to Timo's last post, that could be linked to namespace or symlinks (we don't have any). There's one quite strange thing we've seen. Display the subscription windows in Thunderbird and unchecking INBOX then shows two entries. "INBOX" that isn't selected" and "Inbox" that is selected and cannot been unselected. Checking "INBOX" again makes "Inbox" disappear (subscription window need to be closed & reopened after each action). Now doing some check via telnet doesn't show any mysterious "Inbox"
list "" "*"
- LIST (\HasNoChildren) "." "Sent"
- LIST (\HasNoChildren) "." "Spam"
- LIST (\HasNoChildren) "." "Drafts"
- LIST (\HasNoChildren) "." "Trash"
- LIST (\HasNoChildren) "." "INBOX"
lsub "" "*"
- LSUB () "." "Trash"
- LSUB () "." "Sent"
- LSUB () "." "Spam"
- LSUB () "." "Drafts"
- LSUB () "." "Inbox"
04 select INBOX
- FLAGS (\Answered \Flagged \Deleted \Seen \Draft NonJunk)
- OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft NonJunk \*)] Flags permitted.
- 1 EXISTS
- 0 RECENT
- OK [UIDVALIDITY 1346832313] UIDs valid
- OK [UIDNEXT 82] Predicted next UID
- OK [HIGHESTMODSEQ 51] Highest 04 OK [READ-WRITE] Select completed.
04 select Inbox
- OK [CLOSED] Previous mailbox closed.
- FLAGS (\Answered \Flagged \Deleted \Seen \Draft NonJunk)
- OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft NonJunk \*)] Flags permitted.
- 1 EXISTS
- 0 RECENT
- OK [UIDVALIDITY 1346832313] UIDs valid
- OK [UIDNEXT 82] Predicted next UID
- OK [HIGHESTMODSEQ 51] Highest 04 OK [READ-WRITE] Select completed.
It's working for both but I checked on other working dovecot servers, that seems to be a normal behavior for Inbox folder
I'm not sure that what we see in Thunderbird is related to the problem but who knows. Any help is of course welcome as we're out of ideas :)
-- Sébastien
On 19.9.2012, at 15.15, Cibest ML wrote:
There's one quite strange thing we've seen. Display the subscription windows in Thunderbird and unchecking INBOX then shows two entries. "INBOX" that isn't selected" and "Inbox" that is selected and cannot been unselected. Checking "INBOX" again makes "Inbox" disappear (subscription window need to be closed & reopened after each action).
No idea. INBOX is treated case-insensitively by IMAP protocol.
I'm not sure that what we see in Thunderbird is related to the problem but who knows. Any help is of course welcome as we're out of ideas :)
Create a test server. Make your Dovecot config minimal. Test with another quota backend. Test with local filesystem. See what change breaks the quota calculation. It is somehow related to your current config, since it's not broken for everybody.
Hi Timo, I've finally solved the issue. As usual, thinking out of the box helped :) In fact, it was conf related but not where it was expected. There was an autocreate = Inbox & autosubscribe = Inbox in 90-plugin.conf Having this configuration seems to double count the Inbox quota during quota recalc (but not during normal IMAP operations like moving or receiving a message). Might be some kind of bug somewhere in recalc. The strange thing is that another more simple dovecot system we have (no nfs, all imap/smtp/filtering operations done on the server) also has this plugin conf but doesn't count quota twice. Anyway, after running a small script to recalc all the quota, everything is now fine. Thanks for you help
-- Regards Sébastien
On 19/09/2012 14:47, Timo Sirainen wrote:
On 19.9.2012, at 15.15, Cibest ML wrote:
There's one quite strange thing we've seen. Display the subscription windows in Thunderbird and unchecking INBOX then shows two entries. "INBOX" that isn't selected" and "Inbox" that is selected and cannot been unselected. Checking "INBOX" again makes "Inbox" disappear (subscription window need to be closed & reopened after each action). No idea. INBOX is treated case-insensitively by IMAP protocol.
I'm not sure that what we see in Thunderbird is related to the problem but who knows. Any help is of course welcome as we're out of ideas :) Create a test server. Make your Dovecot config minimal. Test with another quota backend. Test with local filesystem. See what change breaks the quota calculation. It is somehow related to your current config, since it's not broken for everybody.
participants (3)
-
Cibest ML
-
Daniel Parthey
-
Timo Sirainen