[Dovecot] Sendmail and deliver LDA exits with EX_TEMPFAIL on overquota
Hi all,
I have setup dovecot 1.1.4 and sendmail 8.14.3 with dovecot LDA on linux (RHEL5 on x86_64).
I have deliberately set a very low quota (4MB) on a test account, while the actual disk usage is 5MB at the moment, in order to test overquota behaviour..
When I send an email to this account, sendmail accepts the email and tries to deliver it. Deliver fails to save in INBOX and returns an error, but the email stays in sendmail's queue..
Any suggestions on how to avoid this and produce a 5xx error for overquota users before accepting the email would be greatly appreciated..
TIA, Sotiris.
==> /var/log/maillog <== Oct 17 14:25:19 mx-mstr-07 sendmail[17289]: m9HBPJ99017288: to=stsimb@t157.forthnet.gr, ctladdr=root@mx-mstr-07.forthnet.prv (0/0), delay=00:00:00, xdelay=00:00:00, mailer=local, pri=30603, dsn=4.0.0, stat=Deferred: local mailer (/usr/libexec/dovecot/deliver) exited with EX_TEMPFAIL
==> /var/log/dovecot.log <== deliver(stsimb.t157.forthnet.gr): Oct 17 14:25:19 Info: Loading modules from directory: /usr/lib64/dovecot/lda deliver(stsimb.t157.forthnet.gr): Oct 17 14:25:19 Info: Module loaded: /usr/lib64/dovecot/lda/lib10_quota_plugin.so dovecot: Oct 17 14:25:19 Info: auth(default): master in: USER 1 stsimb.t157.forthnet.gr service=deliver dovecot: Oct 17 14:25:19 Info: auth(default): master out: USER 1 stsimb.t157.forthnet.gr uid=2082264764 home=/home/mailusers/stsimb.t157.forthnet.gr gid=126 quota_rule=*:storage=4M deliver(stsimb.t157.forthnet.gr): Oct 17 14:25:19 Info: auth input: stsimb.t157.forthnet.gr deliver(stsimb.t157.forthnet.gr): Oct 17 14:25:19 Info: auth input: uid=2082264764 deliver(stsimb.t157.forthnet.gr): Oct 17 14:25:19 Info: auth input: home=/home/mailusers/stsimb.t157.forthnet.gr deliver(stsimb.t157.forthnet.gr): Oct 17 14:25:19 Info: auth input: gid=126 deliver(stsimb.t157.forthnet.gr): Oct 17 14:25:19 Info: auth input: quota_rule=*:storage=4M deliver(stsimb.t157.forthnet.gr): Oct 17 14:25:19 Info: Quota root: name= backend=maildir args= deliver(stsimb.t157.forthnet.gr): Oct 17 14:25:19 Info: Quota rule: root= mailbox=* bytes=4194304 (0%) messages=0 (0%) deliver(stsimb.t157.forthnet.gr): Oct 17 14:25:19 Info: Quota warning: bytes=3774873 (90%) messages=0 (0%) command=/local/dovecot/sbin/quota-warning.sh 90 deliver(stsimb.t157.forthnet.gr): Oct 17 14:25:19 Info: Quota warning: bytes=2516582 (60%) messages=0 (0%) command=/local/dovecot/sbin/quota-warning.sh 60 deliver(stsimb.t157.forthnet.gr): Oct 17 14:25:19 Info: maildir: data=/var/mail/stsimb.t157.forthnet.gr/Maildir deliver(stsimb.t157.forthnet.gr): Oct 17 14:25:19 Info: maildir++: root=/var/mail/stsimb.t157.forthnet.gr/Maildir, index=, control=, inbox=/var/mail/stsimb.t157.forthnet.gr/Maildir deliver(stsimb.t157.forthnet.gr): Oct 17 14:25:19 Info: msgid=200810171125.m9HBPJFT017287@mx-mstr-07.forthnet.prv: save failed to INBOX: Quota exceeded (mailbox for user is full)
==> sendmail.cf dovecot mailer <== Mlocal, P=/usr/libexec/dovecot/deliver, F=lsDFMAw5:/|@qSPhn9, S=EnvFromL/HdrFromL, R=EnvToL/HdrToL, T=DNS/RFC822/X-Unix, A=deliver -d $u
[root@mx-mstr-07 ~]# dovecot -n # 1.1.4: /etc/dovecot.conf log_path: /var/log/dovecot.log protocols: imap pop3 ssl_disable: yes login_dir: /var/run/dovecot/login login_executable(default): /usr/libexec/dovecot/imap-login login_executable(imap): /usr/libexec/dovecot/imap-login login_executable(pop3): /usr/libexec/dovecot/pop3-login login_greeting: Dovecot ready mx-mstr-07 login_process_per_connection: no login_process_size: 1024 max_mail_processes: 2048 verbose_proctitle: yes mail_location: maildir:/var/mail/%u/Maildir mail_cache_min_mail_count: 100 mail_debug: yes mmap_disable: yes mail_nfs_storage: yes mail_nfs_index: yes maildir_copy_preserve_filename: yes mail_executable(default): /usr/libexec/dovecot/imap mail_executable(imap): /usr/libexec/dovecot/imap mail_executable(pop3): /usr/libexec/dovecot/pop3 mail_plugins(default): quota imap_quota mail_plugins(imap): quota imap_quota mail_plugins(pop3): quota mail_plugin_dir(default): /usr/lib64/dovecot/imap mail_plugin_dir(imap): /usr/lib64/dovecot/imap mail_plugin_dir(pop3): /usr/lib64/dovecot/pop3 pop3_no_flag_updates(default): no pop3_no_flag_updates(imap): no pop3_no_flag_updates(pop3): yes pop3_client_workarounds(default): pop3_client_workarounds(imap): pop3_client_workarounds(pop3): outlook-no-nuls oe-ns-eoh auth default: cache_size: 131072 cache_ttl: 315 cache_negative_ttl: 285 verbose: yes debug: yes passdb: driver: ldap args: /etc/dovecot-ldap.conf userdb: driver: ldap args: /etc/dovecot-ldap.conf userdb: driver: prefetch socket: type: listen master: path: /var/run/dovecot/auth-master mode: 384 plugin: quota: maildir quota_warning: storage=90%% /local/dovecot/sbin/quota-warning.sh 90 quota_warning2: storage=60%% /local/dovecot/sbin/quota-warning.sh 60
On Oct 17, 2008, at 3:39 PM, Sotiris Tsimbonis wrote:
When I send an email to this account, sendmail accepts the email and
tries to deliver it. Deliver fails to save in INBOX and returns an
error, but the email stays in sendmail's queue..Any suggestions on how to avoid this and produce a 5xx error for
overquota users before accepting the email would be greatly
appreciated..
That should be the default. "quota_full_tempfail" setting controls it,
do you have it commented out in the config file? (unfortunately lda
section isn't listed with dovecot -n)
Timo Sirainen wrote, On 10/17/2008 06:37 PM:
On Oct 17, 2008, at 3:39 PM, Sotiris Tsimbonis wrote:
When I send an email to this account, sendmail accepts the email and tries to deliver it. Deliver fails to save in INBOX and returns an error, but the email stays in sendmail's queue..
Any suggestions on how to avoid this and produce a 5xx error for overquota users before accepting the email would be greatly appreciated..
That should be the default. "quota_full_tempfail" setting controls it, do you have it commented out in the config file? (unfortunately lda section isn't listed with dovecot -n)
Sorry, I forgot to include this part.. Unfortunately I've set it to no, that's the problem..
protocol lda { postmaster_address = postmaster@forthnet.gr mail_plugins = quota quota_full_tempfail = no sendmail_path = /usr/sbin/sendmail auth_socket_path = /var/run/dovecot/auth-master }
Sotiris.
On Fri, 2008-10-17 at 22:30 +0300, Sotiris Tsimbonis wrote:
That should be the default. "quota_full_tempfail" setting controls it, do you have it commented out in the config file? (unfortunately lda section isn't listed with dovecot -n)
Sorry, I forgot to include this part.. Unfortunately I've set it to no, that's the problem..
protocol lda { postmaster_address = postmaster@forthnet.gr mail_plugins = quota quota_full_tempfail = no
Well, the problem is again deliver's own stupid configuration parsing. It doesn't know that quota_full_tempfail is a boolean setting, so it treats "no" as "yes". Just remove/comment out the setting entirely and it should work.
http://hg.dovecot.org/dovecot-1.1/rev/f8e6d1922280 also fixes this. And hopefully v1.3 will have rewritten config handling which finally gets rid of these kind of problems.
Timo Sirainen wrote, On 10/18/2008 01:31 PM:
On Fri, 2008-10-17 at 22:30 +0300, Sotiris Tsimbonis wrote:
That should be the default. "quota_full_tempfail" setting controls it, do you have it commented out in the config file? (unfortunately lda section isn't listed with dovecot -n)
Sorry, I forgot to include this part.. Unfortunately I've set it to no, that's the problem..
protocol lda { postmaster_address = postmaster@forthnet.gr mail_plugins = quota quota_full_tempfail = no
Well, the problem is again deliver's own stupid configuration parsing. It doesn't know that quota_full_tempfail is a boolean setting, so it treats "no" as "yes". Just remove/comment out the setting entirely and it should work.
Yeap, it worked now!
deliver(stsimb.t157.forthnet.gr): Oct 18 21:56:30 Info: msgid=200810181856.m9IIu6rc005273@mx-mstr-07.forthnet.gr: save failed to INBOX: Quota exceeded (mailbox for user is full) deliver(stsimb.t157.forthnet.gr): Oct 18 21:56:30 Info: msgid=200810181856.m9IIu6rc005273@mx-mstr-07.forthnet.gr: rejected: Quota exceeded (mailbox for user is full) deliver(stsimb.t157.forthnet.gr): Oct 18 21:56:30 Info: Sending a rejection to stsimb.t157.forthnet.gr: Quota exceeded (mailbox for user is full)
Thanks Timo! Sot.
http://hg.dovecot.org/dovecot-1.1/rev/f8e6d1922280 also fixes this. And hopefully v1.3 will have rewritten config handling which finally gets rid of these kind of problems.
On 10/18/2008 3:04 PM, Sotiris Tsimbonis wrote:
deliver(stsimb.t157.forthnet.gr): Oct 18 21:56:30 Info: msgid=200810181856.m9IIu6rc005273@mx-mstr-07.forthnet.gr: save failed to INBOX: Quota exceeded (mailbox for user is full) deliver(stsimb.t157.forthnet.gr): Oct 18 21:56:30 Info: msgid=200810181856.m9IIu6rc005273@mx-mstr-07.forthnet.gr: rejected: Quota exceeded (mailbox for user is full) deliver(stsimb.t157.forthnet.gr): Oct 18 21:56:30 Info: Sending a rejection to stsimb.t157.forthnet.gr: Quota exceeded (mailbox for user is full)
I'm by no means sure, but this looks like you are BOUNCING a message due to over quota AFTER having already ACCEPTED it... can you confirm/deny this?
--
Best regards,
Charles
Charles Marcus a écrit :
On 10/18/2008 3:04 PM, Sotiris Tsimbonis wrote:
deliver(stsimb.t157.forthnet.gr): Oct 18 21:56:30 Info: msgid=200810181856.m9IIu6rc005273@mx-mstr-07.forthnet.gr: save failed to INBOX: Quota exceeded (mailbox for user is full) deliver(stsimb.t157.forthnet.gr): Oct 18 21:56:30 Info: msgid=200810181856.m9IIu6rc005273@mx-mstr-07.forthnet.gr: rejected: Quota exceeded (mailbox for user is full) deliver(stsimb.t157.forthnet.gr): Oct 18 21:56:30 Info: Sending a rejection to stsimb.t157.forthnet.gr: Quota exceeded (mailbox for user is full)
I'm by no means sure, but this looks like you are BOUNCING a message due to over quota AFTER having already ACCEPTED it... can you confirm/deny this?
yes he does. I hope he has a good spam filter! sending bounces to forged senders isn't a good practice...
mouss wrote, On 10/19/2008 12:50 PM:
Charles Marcus a écrit :
On 10/18/2008 3:04 PM, Sotiris Tsimbonis wrote:
deliver(stsimb.t157.forthnet.gr): Oct 18 21:56:30 Info: msgid=200810181856.m9IIu6rc005273@mx-mstr-07.forthnet.gr: save failed to INBOX: Quota exceeded (mailbox for user is full) deliver(stsimb.t157.forthnet.gr): Oct 18 21:56:30 Info: msgid=200810181856.m9IIu6rc005273@mx-mstr-07.forthnet.gr: rejected: Quota exceeded (mailbox for user is full) deliver(stsimb.t157.forthnet.gr): Oct 18 21:56:30 Info: Sending a rejection to stsimb.t157.forthnet.gr: Quota exceeded (mailbox for user is full) I'm by no means sure, but this looks like you are BOUNCING a message due to over quota AFTER having already ACCEPTED it... can you confirm/deny this?
yes he does. I hope he has a good spam filter! sending bounces to forged senders isn't a good practice...
I confirm, that the above config does produce bounces (possibly forged i.e. backscatter) and it's bad ..
But how do we move the rejection at smtp level using sendmail+dovecot lda?
Sot.
On 10/19/2008, Sotiris Tsimbonis (tsimbonis@forthnet.gr) wrote:
But how do we move the rejection at smtp level using sendmail+dovecot lda?
Thats a sendmail question, so the only answer I can give is...
switch to postfix... ;)
--
Best regards,
Charles
On 10/20/2008 12:57 AM, Charles Marcus wrote:
On 10/19/2008, Sotiris Tsimbonis (tsimbonis@forthnet.gr) wrote:
But how do we move the rejection at smtp level using sendmail+dovecot lda?
Thats a sendmail question, so the only answer I can give is...
switch to postfix... ;)
That's not an acceptable answer ;-))
Cheers, Sotiris.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Sun, 19 Oct 2008, Sotiris Tsimbonis wrote:
But how do we move the rejection at smtp level using sendmail+dovecot lda?
When a recipient is valid, the mail is queued locally, then the delivery attempt is made. That means that the SMTP dialogue has been closed already, before the MDA is invoked. sendmail has quite a strict separation between MTA and MDA, or to put it in other words it supports only the basic Unix-style /var/mail/system_user mboxes natively.
About "Switch to postfix":
How handles postfix the situation, when you mail to an alias with several recipients, one of them is overquota? Is the mail tempfailed for all of them?
==
Well, you could deploy a milter that verifies the recipients at RCPT TO stage. Depending on your config it may be difficult to map recipient addresses to the particular mail storage (esp. if probed on mail relays).
If you have a strict naming policy, you can put all users over quota into the access DB and update them regularily - thus avoding the milter.
Bye,
Steffen Kaiser -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux)
iD8DBQFI/FNkVJMDrex4hCIRAts4AJwKhz7XsnO++lkgGtea2O+gw+RgYQCgq9uD iyG3qlVh0mx4sDTxQTvAqiU= =e/K4 -----END PGP SIGNATURE-----
On 10/20/2008 12:46 PM, Steffen Kaiser wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Sun, 19 Oct 2008, Sotiris Tsimbonis wrote:
But how do we move the rejection at smtp level using sendmail+dovecot lda?
When a recipient is valid, the mail is queued locally, then the delivery attempt is made. That means that the SMTP dialogue has been closed already, before the MDA is invoked. sendmail has quite a strict separation between MTA and MDA, or to put it in other words it supports only the basic Unix-style /var/mail/system_user mboxes natively.
About "Switch to postfix":
How handles postfix the situation, when you mail to an alias with several recipients, one of them is overquota? Is the mail tempfailed for all of them?
==
Well, you could deploy a milter that verifies the recipients at RCPT TO stage. Depending on your config it may be difficult to map recipient addresses to the particular mail storage (esp. if probed on mail relays).
If you have a strict naming policy, you can put all users over quota into the access DB and update them regularily - thus avoding the milter.
I have made such a proof-of-concept milter before for mbox and static quota assignments, see http://stsimb.irc.gr/2008/02/10/milter-quota/ (excuse the Greek language :)).
It has a backend webservice (a cgi script) that calculates and reports "OVERQUOTA" using stat on the mbox, and a small milter on the frontend that queries this webservice for every recepient..
The problem is, that with Maildir it can be very costly to calculate the actual mail usage on every mail delivery attempt..
If I could somehow access dovecot's maildirsize, where the size is already calculated and updated on delivery and retrieval, things would be better..
Sot.
On 10/20/2008 01:11 PM, Sotiris Tsimbonis wrote:
The problem is, that with Maildir it can be very costly to calculate the actual mail usage on every mail delivery attempt..
If I could somehow access dovecot's maildirsize, where the size is already calculated and updated on delivery and retrieval, things would be better..
After reading http://wiki.dovecot.org/Quota/Maildir and the standard Maildir++ specification, it's quite easy to get maildirsize after all..
See http://stsimb.irc.gr/2008/10/20/milter-quota-for-maildir/
So now I can bounce messages using a milter at smtp level :)
Sot.
Steffen Kaiser skdovecot@smail.inf.fh-brs.de wrote on 20 Oct 2008 11:46:
On Sun, 19 Oct 2008, Sotiris Tsimbonis wrote:
But how do we move the rejection at smtp level using sendmail+dovecot lda?
When a recipient is valid, the mail is queued locally, then the delivery attempt is made. That means that the SMTP dialogue has been closed already, before the MDA is invoked. sendmail has quite a strict separation between MTA and MDA, or to put it in other words it supports only the basic Unix-style /var/mail/system_user mboxes natively.
I'm not sure how sendmail handles delivery via LMTP. _May be_ this can improve the situation. Of course, dovecot's deliver must support it.
-- Frank Behrens, Osterwieck, Germany PGP-key 0x5B7C47ED on public servers available.
Sotiris Tsimbonis a écrit :
I confirm, that the above config does produce bounces (possibly forged i.e. backscatter) and it's bad ..
But how do we move the rejection at smtp level using sendmail+dovecot lda?
for that you need to detect over quota during the smtp transaction, but in general, MTAs queue mail first, and deliver later.
There is no "perfect" way to deal with this issue. even if you use a milter (or policy server or proxy_filter...), you will find it hard to take into account mail that is still in the MTA queue (for that, you would need to detect which mail was delivered since you started counting, ... etc). really not something you should go for.
You can however mitigate the problem.
you can populate an access list to reject mail to users who are over quota. you need some way to remove them from the list once they purge their mailbox (web ui, cron, dovecot plugin, ...).
filter spam (as much as you can) and don't bounce if the message is spam. The issue here is what to do with the message (because of FPs, you can't simply discard it). on the other hand, you can reject junk transactions at smtp time (well, at least some volume of junk)
My preference for quota handling is this:
- user has two quotas. say 100 Mo and 150 Mo.
- if he reaches 100 Mo, he is warned. he should purge his mailbox.
- if he reaches 150 Mo, his address is blocked. He will need to purge his mailbox and ask to be delisted (maybe via a web UI).
- when mail is received, users who are between 100 and 150 are checked in real time. if this shows them reaching 150, they are added to the "block list". otherwise, mail is delivered as usual.
with this, no real time quota checks are done for users under the "low" threshold. if most of your users fall in this category, this means that real time checks are relatively rare.
mouss wrote, On 10/20/2008 05:04 PM:
Sotiris Tsimbonis a écrit : their mailbox (web ui, cron, dovecot plugin, ...).
- you can populate an access list to reject mail to users who are over quota. you need some way to remove them from the list once they purge
[...]
My preference for quota handling is this:
- user has two quotas. say 100 Mo and 150 Mo.
- if he reaches 100 Mo, he is warned. he should purge his mailbox.
- if he reaches 150 Mo, his address is blocked. He will need to purge his mailbox and ask to be delisted (maybe via a web UI).
- when mail is received, users who are between 100 and 150 are checked in real time. if this shows them reaching 150, they are added to the "block list". otherwise, mail is delivered as usual.
Very interesting idea..
I could populate a list with users approaching quota (over a certain threshold) and use it in my milter in order to avoid calling the backend quota-reporting service for every user..
Cheers, Sotiris.
participants (6)
-
Charles Marcus
-
Frank Behrens
-
mouss
-
Sotiris Tsimbonis
-
Steffen Kaiser
-
Timo Sirainen