quota-status fails when size=0 from Postfix
Dear list,
I've been pulling my hair out today and kindly ask you for help. I'm on Debian Bullseye with Dovecot 2.3.13 and Postfix 3.5.6. Users are stored in SQL and I want to use quotas. Basically I followed https://doc.dovecot.org/configuration_manual/quota_plugin/#quota-service
With mail_debug enabled I see that the information from SQL is fetched correctly:
dovecot: quota-status(test@bullseye.example.org)<45148><>: Debug: auth-master: userdb lookup(test@bullseye.example.org): auth USER input: test@bullseye.example.org quota_rule=*:bytes=1000 home=/var/vmail/bullseye.example.org/test uid=5000 gid=5000
So the quota rule "*:bytes=1000" is loaded as intended for testing. Before handing the email over to Dovecot through LMTP Postfix should call the quota-service (policy daemon) to determine if the user is over quota:
smtpd_recipient_restrictions =
reject_unauth_destination
check_policy_service inet:localhost:12340
This is still by the book.
What I found:
- sending a test mail with Thunderbird properly returns a 5xx status (over quota)
- sending a test mail with SWAKS returns DUNNO (swaks is an SMTP test tool)
I have used tcpdump to look into the communication on port 12340 and the difference appears to be the "size=0" line that Postfix sends if I use SWAKS. Output from SWAKS: -> RCPT TO:test@bullseye.example.org <- 250 2.1.5 Ok
However if Thunderbird sends an email it uses something like
-> RCPT TO:test@bullseye.example.org SIZE=12345
If I understand the situation correctly then Postfix extracts the SIZE=… value and uses that to send it to the quota-status policy daemon.
The user however is definitely over quota:
$> doveadm quota get -u test@bullseye.example.org Quota name Type Value Limit % User quota STORAGE 1 1 100 User quota MESSAGE 1 - 0
So whatever the size of the incoming email is, it should be rejected. As quota-status returns DUNNO, Postfix will send the email to Dovecot through LMTP. And LMTP in turn does another quota check and correctly rejects the email:
postfix/lmtp[45153]: 573419D6A3: to=test@bullseye.example.org, relay=iredmail-test[private/dovecot-lmtp], delay=0.08, delays=0.05/0.01/0.01/0.01, dsn=5.2.2, status=bounced (host iredmail-test[private/dovecot-lmtp] said: 552 5.2.2 test@bullseye.example.org Quota exceeded (mailbox for user is full) (in reply to end of DATA command))
Problem here is that a bounce is created and Postfix is sending backscatter. Had "quota-service" indicated that the mailbox is over quota then Postfix had instantly rejected the email without any bounce.
I can also reproduce this effect by talking to the quota-status daemon:
printf "recipient=test@bullseye.example.org\nsize=0\n\n" | nc localhost 12340 action=DUNNO
printf "recipient=test@bullseye.example.org\nsize=10000\n\n" | nc localhost 12340 action=554 5.2.2 Quota exceeded (mailbox for user is full)
Is this a feature? A bug? An evil coincidence?
Thanks for reading this far. I know that this matter seems a bit complicated. But it feels like Dovecot's behavior leads to backscatter and of course I'd like to avoid that.
Kindly… Christoph
- sending a test mail with SWAKS returns DUNNO (swaks is an SMTP test tool)
the difference appears to be the "size=0" line that Postfix sends if I use SWAKS.
The postfix policy server sends over a bunch of key=values when it runs. It has no idea why its running, it doesn't know you want it to be checking for over quota. It just sends every thing over to dovcot who knows that you want to be checking for over quota. One of the key=values postfix sends is size=<bytes> of the message size. Dovecot looks at this and ignores the rest of what postfix sent and returns a result.
Now im guessing, maybe because you are using a testing tool, maybe it is not sending a message body in the email transmission, so postfix is correcting reporting that message has an email body of zero bytes (size=0). Dovecot must approve this because current maildir size + 0 != over limit. Maybe you are setup correctly and its just the testing tool that creates an edge case you wont experience in the wild.
Like you said when sending a real email it gets rejected right?
Am 29.07.21 um 02:44 schrieb dovecot@ptld.com:
- sending a test mail with SWAKS returns DUNNO (swaks is an SMTP test tool)
the difference appears to be the "size=0" line that Postfix sends if I use SWAKS.
Now im guessing, maybe because you are using a testing tool, maybe it is not sending a message body in the email transmission, so postfix is correcting reporting that message has an email body of zero bytes (size=0).
There is a little body reading "This is a test mailing".
Dovecot must approve this because current maildir size + 0 != over limit. Maybe you are setup correctly and its just the testing tool that creates an edge case you wont experience in the wild.
That may totally be an edge case. But even then
over-quota + 0 = still-over-quota
Like you said when sending a real email it gets rejected right?
At least from Thunderbird, yes. But SIZE=… does not seem to be a mandatory addition to the RCPT-TO line. Perhaps other mail clients do not use SIZE either. Just guessting.
Thanks for your thoughts.
…Christoph
At least from Thunderbird, yes. But SIZE=… does not seem to be a mandatory addition to the RCPT-TO line. Perhaps other mail clients do not use SIZE either. Just guessting.
http://www.postfix.org/SMTPD_POLICY_README.html The "size" attribute value specifies the message size that the client specified in the MAIL FROM command (zero if none was specified). With Postfix 2.2 and later, it specifies the actual message size after the client sends the END-OF-MESSAGE.
Okay, it looks like if you are using an old version of postfix then postfix does rely on the client telling it the size of the email. But with postfix version 2.2 or newer postfix computes the size itself.
At least from Thunderbird, yes. But SIZE=… does not seem to be a mandatory addition to the RCPT-TO line. Perhaps other mail clients do not use SIZE either. Just guessting. Correct. It's optional and can't be trusted anyway.
http://www.postfix.org/SMTPD_POLICY_README.html The "size" attribute value specifies the message size that the client specified in the MAIL FROM command (zero if none was specified). With Postfix 2.2 and later, it specifies the actual message size after the client sends the END-OF-MESSAGE.
<bad idea>
I misread that and Wietse (Postfix developer) kindly pointed me to the "after" part. So I found an evil way to make it work:
smtpd_recipient_restrictions = reject_unauth_destination
smtpd_end_of_data_restrictions =
check_policy_service inet:localhost:12340
The smtpd_end_of_data_restrictions are evaluated after the complete email was received. Of course this is wasteful because receiving a large email only to reject it afterwards is not very considerate.
Plus Dovecot complains that the policy service is only supposed to be used in the RCPT stage. So clearly this is a bad approach.
In my opinion the quota-status service from Dovecot should be able to reject any email no matter the size if the user is over quota. But I can't get that to work. I have set quota_grace to 0. And the user is using 100% quota. Still…
printf "recipient=test@bullseye.example.org\nsize=100\n\n" |
nc localhost 12340
action=DUNNO
printf "recipient=test@bullseye.workaround.org\nsize=1000\n\n" |
nc localhost 12340
action=554 5.2.2 Quota exceeded (mailbox for user is full)
Dovecot is still responding with DUNNO. It accepts size=0. It accepts size=100. Only at a much larger size (1000) it sends a rejection. Why is that?
In https://github.com/dovecot/core/blob/master/src/plugins/quota/quota-status.c there are several e_debug statements. How do I enable that debug logging for the quota plugin? Maybe that gives an idea.
I really wouldn't want to script my own policy daemon just to avoid backscatter. :(
Thanks in advance… …Christoph
Plus Dovecot complains that the policy service is only supposed to be used in the RCPT stage. So clearly this is a bad approach.
I want to explore this more. I tried it and also see:
dovecot[1096]: quota-status(26164): Warning: Received policy query from MTA in unexpected state END-OF-MESSAGE (service can only be used for recipient restrictions)
Why? Why does dovecot even care? Quota plugin is sent a user and a size, it looks up quota for that user and computes if size will put the user over limit and returns an answer. Why does dovecot care or even know at what stage this is done? Why is it bad to check quota after getting the real size? Seems like its designed to allow spoofing from an evil mail client.
What is the harm being done that causes this log warning? What is the harm in ignoring the warning?
On 7/29/2021 2:15 PM, dovecot@ptld.com wrote:
Plus Dovecot complains that the policy service is only supposed to be used in the RCPT stage. So clearly this is a bad approach.
I want to explore this more. I tried it and also see:
dovecot[1096]: quota-status(26164): Warning: Received policy query from MTA in unexpected state END-OF-MESSAGE (service can only be used for recipient restrictions)
Why? Why does dovecot even care? Quota plugin is sent a user and a size, it looks up quota for that user and computes if size will put the user over limit and returns an answer. Why does dovecot care or even know at what stage this is done? Why is it bad to check quota after getting the real size? Seems like its designed to allow spoofing from an evil mail client.
What is the harm being done that causes this log warning? What is the harm in ignoring the warning?
With multi-recipient mail, the recipient attribute is undefined at end-of-data.
So you have to pick your poison - during recipient restrictions the size may not be known or may not be accurate, at end-of-data the recipient may not be known *and* it's too late to reject a single recipient on a multi-recipient mail.
The only solution is to reject all mail for an over-quota recipient during recipient restrictions, and if the mail passes that stage, deliver it anyway even if it makes the user go over quota.
-- Noel Jones
On 2021-07-29 21:34, Christoph Haas wrote:
Noel:
The only solution is to reject all mail for an over-quota recipient during recipient restrictions, and if the mail passes that stage, deliver it anyway even if it makes the user go over quota.
Seconded. Is that possible with the current quota-status?
yes, and even it works with dokumented default config
what does not work is the size=forged-size problem is not implented since it only works in postfix eof-data stage in mta, that means we would all accept iso file sizes before we discovery that its oversizes in the first place
dovecot will tell postfix to reject over quotas based on what is free left quotas
it cant be solved better not even in bash
but one thing, it could be changed to use dovecot dict: if this is already existing, then postfix could have a policy server that checks dict: maps, and basicly this will provide support for dovecot can be stopped but postfix still use dict in all supported postconf -m table supported, if dovecot support them all
but the current implemention is imho still better
The only solution is to reject all mail for an over-quota recipient during recipient restrictions, and if the mail passes that stage, deliver it anyway even if it makes the user go over quota.
But does this happen out of the box? Wont dovecot end up back splatter bouncing one of the recipients that is over quota after postfix delivers all recipients to LMTP?
On 7/29/2021 2:44 PM, dovecot@ptld.com wrote:
The only solution is to reject all mail for an over-quota recipient during recipient restrictions, and if the mail passes that stage, deliver it anyway even if it makes the user go over quota.
But does this happen out of the box? Wont dovecot end up back splatter bouncing one of the recipients that is over quota after postfix delivers all recipients to LMTP?
That's probably what happens now when user not over quota and receives a large mail that would put them over quota.
What is needed is up-to-quota-plus-one-more-mail-regardless-of-size.
MTA- I have a 0 byte mail for bob. Is he over quota? D- nope, he's good for another 50Kb. MTA- Ok, here's the mail, it's really 3Mb. D- Thanks, delivered.
-- next run --
MTA- I have a 0 byte mail for bob. Is he over quota? D- Yes. MTA- Thanks, rejected.
-- Noel Jones
On 2021-07-29 20:13, Christoph Haas wrote:
I really wouldn't want to script my own policy daemon just to avoid backscatter. :(
why not ? :=)
quota grace is ideal with same size as max mailsize in postfix, with is 10MB as standard, the quota grace ensure one mail is possible to be sent if its not much mail left in dovecot to fill the quotas 100%
so setting it to 0 is silly like set non default settings in postfix main.cf, limits are there for something
Am 29.07.21 um 22:08 schrieb Benny Pedersen:
On 2021-07-29 20:13, Christoph Haas wrote:
I really wouldn't want to script my own policy daemon just to avoid backscatter. :(
why not ? :=)
I'm considering it. But I still hope that I'm doing something wrong or that Dovecot's plugin can be improved.
quota grace is ideal with same size as max mailsize in postfix, with is 10MB as standard, the quota grace ensure one mail is possible to be sent if its not much mail left in dovecot to fill the quotas 100%
so setting it to 0 is silly like set non default settings in postfix main.cf, limits are there for something
I don't mind using the grace amount. But I don't want to accept that the quota-status tells me that the user is under quota only to be rejected in the next step with LMTP.
On 2021-07-29 02:44, dovecot@ptld.com wrote:
Like you said when sending a real email it gets rejected right?
postfix can only check results from dovecot, not dictate that new mails size is a real iso or other content, mta stage cant be made to hand over size in policy server since email then must send whole email first, and that require more bandwidth for all parties to do this, so it just checking dovecot quotas to find if there is room for one more email in a giving quotas, if its still room for more mails i send msg to postfix accept it, but when its not more quotas in dovecot result is changed to reject, postfix does not still know how much quotas is in dovecot
hopefully the above make sense then
quotas should imho just be disabled for sasl users, that is missing as an option on dovecot quota services
and since dovecot does not know postfix virtual aliases is would make sense dovecot also support postfix virtual aliases in dovecot quota service, or postfix have to map alias before sending to lmtp server on dovecot
and since dovecot does not know postfix virtual aliases is would make sense dovecot also support postfix virtual aliases in dovecot quota service
Why not? You can create a query telling dovecot to lookup the quota of the mailbox the alias points to. Ofcourse your tables wont look like mine but this example can give you an idea of how to set it up.
SELECT accounts.inbox AS user, CONCAT('*:storage=', accounts.diskQuota, 'm') AS quota_rule FROM aliases LEFT JOIN accounts USING(email) WHERE aliases.alias = '%u' AND accounts.email IS NOT NULL LIMIT 1
You setup multiple userdb{} as fall backs. If your first query looking up email accounts doesn't find a match it processes the next userdb{} in line which would be a query looking for a match in aliases.
participants (4)
-
Benny Pedersen
-
Christoph Haas
-
dovecot＠ptld.com
-
Noel