Quota status to postfix in distributed environment
Hi
I have implemented Quota status to postfix in our setup. I have an imap server (dovecot) and mail server (postfix) in every node. I am able to send quota status to postfix and mails are rejected after 100% mail quota is crossed. This rejection is happening both in across the nodes and within the nodes.
The problem is if I am sending mails to any node and if any other node's dovecot is down, mails are not going. For example, I am sending an email within the system but if some other node's dovecot is down then email within the system also will not go.
My dovecot version is 2.2.10. My postfix version is 2.1.10.
*doveconf -n output is below:-*
# 2.2.10: /etc/dovecot/dovecot.conf # OS: Linux 3.10.0-514.el7.x86_64 x86_64 Red Hat Enterprise Linux Server release 7.3 (Maipo) xfs auth_debug = yes base_dir = /var/run/dovecot/ first_valid_gid = 5000 first_valid_uid = 5000 hostname = CmdHQ login_greeting = ^^^^^^^^^^Dovecot ready^^^^^^^^^^ mail_debug = yes mail_gid = 6000 mail_location = Maildir:/var/mail/vmail/tcs.mil.in/%n mail_plugins = " quota" mail_uid = 6000 mbox_write_locks = fcntl passdb { args = /etc/dovecot/dovecot-ldap.conf driver = ldap } plugin { quota = maildir:User quota quota_rule = *:storage=8KB quota_rule2 = *:messages=12B quota_status_nouser = DUNNO quota_status_overquota = 552 5.2.2 Mailbox is over quota / mailbox is full quota_status_success = DUNNO quota_warning = storage=80%% quota-warning 80 %u } postmaster_address = postmaster@tcs.mil.in service auth { unix_listener auth-userdb { mode = 0600 user = postfix } } service lmtp { unix_listener /var/spool/postfix/private/dovecot-lmtp { group = postfix mode = 0600 user = postfix } } service quota-status { client_limit = 1 executable = quota-status -p postfix inet_listener { port = 54317 } } service quota-warning { executable = script /usr/local/bin/quota-warning.sh unix_listener quota-warning { group = postfix mode = 0666 user = postfix } user = postfix } ssl = required ssl_ca = </etc/dovecot/certs/cacert.pem ssl_cert = </etc/dovecot/certs/1CorpHQ_IMAP_Admin@tcs.mil.in.pem ssl_key = </etc/dovecot/certs/1CorpHQ_IMAP_Admin@tcs.mil.in.key userdb { args = /etc/dovecot/dovecot-ldap.conf driver = ldap } verbose_ssl = yes protocol lmtp { info_log_path = /var/log/dovecot-lmtp.log mail_plugins = " quota" } protocol lda { info_log_path = /var/log/dovecot-lda.log log_path = /var/log/dovecot-lda-errors.log mail_plugins = " quota" } protocol imap { mail_plugins = " quota" }
Here "service quota status" is the concerned section in conf file.
*Postfix configuration is below:- *
smtpd_relay_restrictions = check_policy_service inet:201.123.80.9:54317 check_policy_service inet:201.123.80.23:54317
virtual_transport=lmtp:unix:private/dovecot-lmtp
Here, I am querying both two nodes. 201.123.80.9 is the other node. 201.123.80.23 is the node within which, email is sent.
*logs while sending mail is below:-F*eb 22 12:43:24 1CorpHQ postfix/proxymap[7327]: In dict_changed_name Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: initializing the server-side TLS engine Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: In dict_changed_name Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: match_list_match: 1CorpHQ: no match Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: match_list_match: 201.123.80.23: no match Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: match_list_match: 1CorpHQ: no match Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: match_list_match: 201.123.80.23: no match Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: text 220 1CorpHQserver.tcs.mil.in ESMTP Postfix Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: text EHLO 1CorpHQ Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: match_list_match: 1CorpHQ: no match Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: match_list_match: 201.123.80.23: no match Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: text 250-1CorpHQserver.tcs.mil.in Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: text 250-PIPELINING Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: text 250-SIZE 10240000 Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: text 250-VRFY Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: text 250-ETRN Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: text 250-STARTTLS Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: text 250-ENHANCEDSTATUSCODES Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: text 250-8BITMIME Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: text 250 DSN Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: text STARTTLS Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: text 220 2.0.0 Ready to start TLS Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: setting up TLS connection from 1CorpHQ[201.123.80.23] Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: 1CorpHQ[201.123.80.23]: TLS cipher list "aNULL:-aNULL:ALL:!EXPORT:!LOW:+RC4:@STRENGTH" Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: SSL_accept:before/accept initialization Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: SSL_accept:SSLv3 read client hello A Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: SSL_accept:SSLv3 write server hello A Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: SSL_accept:SSLv3 write certificate A Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: SSL_accept:SSLv3 write key exchange A Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: SSL_accept:SSLv3 write server done A Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: SSL_accept:SSLv3 flush data Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: SSL_accept:SSLv3 read client key exchange A Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: SSL_accept:SSLv3 read finished A Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: SSL_accept:SSLv3 write change cipher spec A Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: SSL_accept:SSLv3 write finished A Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: SSL_accept:SSLv3 flush data Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: Anonymous TLS connection established from 1corphq[201.123.80.23]: TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits) Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: text EHLO 1CorpHQ Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: match_list_match: 1CorpHQ: no match Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: match_list_match: 201.123.80.23: no match Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: text 250-1CorpHQserver.tcs.mil.in Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: text 250-PIPELINING Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: text 250-SIZE 10240000 Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: text 250-VRFY Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: text 250-ETRN Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: text 250-ENHANCEDSTATUSCODES Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: text 250-8BITMIME Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: text 250 DSN Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: text MAIL FROM:< Cdr.1CorpHQ@tcs.mil.in> Feb 22 12:43:24 1CorpHQ postfix/trivial-rewrite[7330]: match_list_match: transport_maps: no match Feb 22 12:43:24 1CorpHQ postfix/trivial-rewrite[7330]: match_list_match: transport_maps: no match Feb 22 12:43:24 1CorpHQ postfix/trivial-rewrite[7330]: In dict_changed_name Feb 22 12:43:24 1CorpHQ postfix/trivial-rewrite[7330]: match_list_match: tcs.mil.in: no match Feb 22 12:43:24 1CorpHQ postfix/trivial-rewrite[7330]: match_list_match: tcs.mil.in: no match Feb 22 12:43:24 1CorpHQ postfix/trivial-rewrite[7330]: match_list_match: tcs.mil.in: no match Feb 22 12:43:24 1CorpHQ postfix/trivial-rewrite[7330]: match_list_match: tcs.mil.in: no match Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: In valid verify sender addr Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: text 250 2.1.0 Ok Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: text RCPT TO:< CO.1CorpHQ@tcs.mil.in> Feb 22 12:43:24 1CorpHQ postfix/trivial-rewrite[7330]: match_list_match: tcs.mil.in: no match Feb 22 12:43:24 1CorpHQ postfix/trivial-rewrite[7330]: match_list_match: tcs.mil.in: no match Feb 22 12:43:24 1CorpHQ postfix/trivial-rewrite[7330]: match_list_match: tcs.mil.in: no match Feb 22 12:43:24 1CorpHQ postfix/trivial-rewrite[7330]: match_list_match: tcs.mil.in: no match Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: In valid verify sender addr Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: match_list_match: permit_mynetworks: no match Feb 22 12:43:24 1CorpHQ dovecot: quota-status: Debug: Loading modules from directory: /usr/lib64/dovecot Feb 22 12:43:24 1CorpHQ dovecot: quota-status: Debug: Module loaded: /usr/lib64/dovecot/lib10_quota_plugin.so Feb 22 12:43:24 1CorpHQ dovecot: auth: Debug: Loading modules from directory: /usr/lib64/dovecot/auth Feb 22 12:43:24 1CorpHQ dovecot: auth: Debug: Module loaded: /usr/lib64/dovecot/auth/libdriver_sqlite.so Feb 22 12:43:24 1CorpHQ dovecot: auth: Debug: Loading modules from directory: /usr/lib64/dovecot/auth Feb 22 12:43:24 1CorpHQ dovecot: auth: Debug: Module loaded: /usr/lib64/dovecot/auth/libauthdb_ldap.so Feb 22 12:43:24 1CorpHQ dovecot: auth: Debug: Read auth token secret from /var/run/dovecot//auth-token-secret.dat Feb 22 12:43:24 1CorpHQ dovecot: auth: Debug: master in: USER#0111# 011CO.1CorpHQ@tcs.mil.in#011service=quota-status Feb 22 12:43:24 1CorpHQ dovecot: auth: Debug: ldap(co.1corphq@tcs.mil.in): user search: base=dc=tcs,dc=mil,dc=in scope=subtree filter=(&(objectClass=person)(uid=co.1corphq)) fields=homeDirectory,uidNumber,gidNumber Feb 22 12:43:24 1CorpHQ dovecot: auth: Debug: ldap(co.1corphq@tcs.mil.in): no fields returned by the server Feb 22 12:43:24 1CorpHQ dovecot: auth: Debug: ldap(co.1corphq@tcs.mil.in): result: homeDirectory missing; uidNumber missing; gidNumber missing Feb 22 12:43:24 1CorpHQ dovecot: auth: Debug: userdb out: USER#0111# 011co.1corphq@tcs.mil.in Feb 22 12:43:24 1CorpHQ dovecot: quota-status: Debug: auth input: co.1corphq@tcs.mil.in Feb 22 12:43:24 1CorpHQ dovecot: quota-status: Debug: changed username to co.1corphq@tcs.mil.in Feb 22 12:43:24 1CorpHQ dovecot: quota-status: Debug: Added userdb setting: plugin/=yes Feb 22 12:43:24 1CorpHQ dovecot: quota-status(co.1corphq@tcs.mil.in): Debug: Effective uid=6000, gid=6000, home= Feb 22 12:43:24 1CorpHQ dovecot: quota-status(co.1corphq@tcs.mil.in): Debug: Quota root: name=User quota backend=maildir args= Feb 22 12:43:24 1CorpHQ dovecot: quota-status(co.1corphq@tcs.mil.in): Debug: Quota rule: root=User quota mailbox=* bytes=8192 messages=0 Feb 22 12:43:24 1CorpHQ dovecot: quota-status(co.1corphq@tcs.mil.in): Debug: Quota rule: root=User quota mailbox=* bytes=8192 messages=12 Feb 22 12:43:24 1CorpHQ dovecot: quota-status(co.1corphq@tcs.mil.in): Debug: Quota warning: bytes=6553 (80%) messages=0 reverse=no command=quota-warning 80 co.1corphq@tcs.mil.in Feb 22 12:43:24 1CorpHQ dovecot: quota-status(co.1corphq@tcs.mil.in): Debug: Quota grace: root=User quota bytes=819 (10%) Feb 22 12:43:24 1CorpHQ dovecot: quota-status(co.1corphq@tcs.mil.in): Debug: maildir++: root=/var/mail/vmail/tcs.mil.in/co.1corphq, index=, indexpvt=, control=, inbox=/var/mail/vmail/tcs.mil.in/co.1corphq, alt= Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: warning: connect to 201.123.80.9:54317: Connection refused Feb 22 12:43:24 1CorpHQ postfix/smtpd[7326]: warning: problem talking to server 201.123.80.9:54317: Connection refused Feb 22 12:43:25 1CorpHQ postfix/smtpd[7326]: warning: connect to 201.123.80.9:54317: Connection refused Feb 22 12:43:25 1CorpHQ postfix/smtpd[7326]: warning: problem talking to server 201.123.80.9:54317: Connection refused Feb 22 12:43:25 1CorpHQ postfix/smtpd[7326]: NOQUEUE: reject: RCPT from 1CorpHQ[201.123.80.23]: 451 4.3.5 Server configuration problem; from=< Cdr.1CorpHQ@tcs.mil.in> to=<CO.1CorpHQ@tcs.mil.in> proto=ESMTP helo=<1CorpHQ> Feb 22 12:43:25 1CorpHQ postfix/smtpd[7326]: text 451 4.3.5 Server configuration problem Feb 22 12:43:25 1CorpHQ postfix/smtpd[7326]: text RSET Feb 22 12:43:25 1CorpHQ postfix/smtpd[7326]: text 250 2.0.0 Ok Feb 22 12:43:25 1CorpHQ postfix/smtpd[7326]: lost connection after RSET from 1CorpHQ[201.123.80.23]
I am understanding what the logs are trying to say. But I am not able to resolve the issue even after searching solution on internet and trying different hit and trials by myself. I want that if i am sending email to any node or within node, the configuration relating to "check _policy_service" for other node does not interfere and mail goes properly. At the same time I can also fetch quota status from other nodes.
If I can get any help regarding this it will be really appreciable as I have tried a lot of options already.
Regards
On 2018-02-26 16:28, SAAHIL IFTEKHAR wrote:
Hi
I have implemented Quota status to postfix in our setup. I have an imap server (dovecot) and mail server (postfix) in every node. I am able to send quota status to postfix and mails are rejected after 100% mail quota is crossed. This rejection is happening both in across the nodes and within the nodes.
The problem is if I am sending mails to any node and if any other node's dovecot is down, mails are not going. For example, I am sending an email within the system but if some other node's dovecot is down then email within the system also will not go.
My dovecot version is 2.2.10. My postfix version is 2.1.10.
doveconf -n output is below:-
[snap]
Here "service quota status" is the concerned section in conf file.
Postfix configuration is below:- smtpd_relay_restrictions = check_policy_service inet:201.123.80.9:54317 check_policy_service inet:201.123.80.23:54317
virtual_transport=lmtp:unix:private/dovecot-lmtp
Here, I am querying both two nodes. 201.123.80.9 is the other node. 201.123.80.23 is the node within which, email is sent.
logs while sending mail is below:-
[snap]
I am understanding what the logs are trying to say. But I am not able to resolve the issue even after searching solution on internet and trying different hit and trials by myself. I want that if i am sending email to any node or within node, the configuration relating to "check _policy_service" for other node does not interfere and mail goes properly. At the same time I can also fetch quota status from other nodes.
If I can get any help regarding this it will be really appreciable as I have tried a lot of options already.
Your problem is that if the dovecot is unreachable Postfix can't check the quota for the user.
http://www.postfix.org/SMTPD_POLICY_README.html explains how to resolve this problem: smtpd_policy_service_default_action (default: 451 4.3.5 Server configuration problem): The default action when an SMTPD policy service request fails. Available with Postfix 3.0 and later.
The default action when an SMTPD policy service request fails. Specify "DUNNO" to behave as if the failed SMTPD policy service request was not sent, and to continue processing other access restrictions, if any. You can configure it per policy also.
The issue you might encounter with this setup is that if both of your dovecot nodes are unreachable Postfix will accept the e-mail and try to deliver it. When the nodes come online it will fail if the user is over quota and generate bounce to the envelope sender of the message, which might produce backscatter.
I resolved this problem by keeping quota information in Mysql table and using custom policy server to check if user is allowed to receive more e-mail.
Also this approach might not work with aliases and other redirections, the e-mail address checked by smtpd policy is the one before alias expansion. You should check if it suits your environment.
Best, Karol
-- Karol Augustin karol@augustin.pl http://karolaugustin.pl/ +353 85 775 5312
Out of curiosity would it be possible to configure dovecot in such a way that if a message had been accepted by postfix because it couldn't check quota the message was delivered anyway, despite being over quota?
I'd much rather that then risk backscatter.
(I used to have quotas, but with the size of disks I've found it easier to simply have a "clue bat”approach and a reasonable max-message size. When a user gets "too much”mail ask them to purge their older mails with attachments.
Granted, that's probably less workable than quotas for a large user base, but for a period of time I did automate these notices by listing users total storage and sending a heads-up message when they got over a specific number. Granted, that was in the days where a 100MB mail store was a reasonable limit.
-- My main job is trying to come up with new and innovative and effective ways to reject even more mail. I'm up to about 97% now.
On Feb 27, 2018, at 06:18, Karol Augustin <karol@augustin.pl> wrote:
The issue you might encounter with this setup is that if both of your dovecot nodes are unreachable Postfix will accept the e-mail and try to deliver it. When the nodes come online it will fail if the user is over quota and generate bounce to the envelope sender of the message, which might produce backscatter.
I resolved this problem by keeping quota information in Mysql table and using custom policy server to check if user is allowed to receive more e-mail.
On 2018-02-27 15:26, LuKreme wrote:
Out of curiosity would it be possible to configure dovecot in such a way that if a message had been accepted by postfix because it couldn't check quota the message was delivered anyway, despite being over quota? There is quota grace parameter, but it will work for the first email AFAIK.
In properly configured setup there should be no risk of Dovecot being inaccessible.
Anyway, I still believe that configuring Dovecot to store quota in mysql and quering that using policy server is the best approach. Especially if mysql server is properly replicated and always accessible. In my case mysql is required for e-mail delivery anyways (aliases, users, passwords, webmail etc).
Since OP didn't share his requirements regarding other parts of the system there is not much that can be suggested to help. Especially that it looks like he is forced to work with ancient versions of software, which usually happens in environments where management doesn't listen to architecture suggestions anyway.
I'd much rather that then risk backscatter.
(I used to have quotas, but with the size of disks I've found it easier to simply have a "clue bat”approach and a reasonable max-message size. When a user gets "too much”mail ask them to purge their older mails with attachments.
This is reasonable approach however there is remote possibility that if user gets compromised someone might fill their account (and your disk) using IMAP killing your server.
Granted, that's probably less workable than quotas for a large user base, but for a period of time I did automate these notices by listing users total storage and sending a heads-up message when they got over a specific number. Granted, that was in the days where a 100MB mail store was a reasonable limit.
-- My main job is trying to come up with new and innovative and effective ways to reject even more mail. I'm up to about 97% now.
On Feb 27, 2018, at 06:18, Karol Augustin <karol@augustin.pl> wrote:
The issue you might encounter with this setup is that if both of your dovecot nodes are unreachable Postfix will accept the e-mail and try to deliver it. When the nodes come online it will fail if the user is over quota and generate bounce to the envelope sender of the message, which might produce backscatter.
I resolved this problem by keeping quota information in Mysql table and using custom policy server to check if user is allowed to receive more e-mail.
-- Karol Augustin karol@augustin.pl http://karolaugustin.pl/ +353 85 775 5312
participants (4)
-
＠lbutlr
-
Karol Augustin
-
LuKreme
-
SAAHIL IFTEKHAR