Unable to get quotas working
Ubuntu Version: Linux 6.2.0-1017-aws x86_64 Ubuntu 22.04.4 LTS nfs4
Postfix: Version: 3.6.4
Dovecot Version: 2.3.16 (7e2e900c1a)
Pigeonhole Version: 0.5.16 (09c29328)
Rspamd Version: 3.8.4
Protocols: IMAP, LMTP, SMTP
Setup: I have an email server running with virtual domains and virtual mailboxes that is currently hosting about 10GB of email across 15 domains and 8 mailboxes which will expand drastically once I get this to work. The server is an AWS EC2 instance with an AWS RDS instance for the database and using AWS EFS for storage. Everything works quite well except quotas!
Problem: I am having an issue with getting quotas to work. When using “doveadm”, I can list the mailboxes however the “Limit” column is just a hyphen “-“ like there is no limit even though every account is set to 10GB for their limit in the database. At the same time, the actual used storage and messages is not getting updated in the database either yet there are no errors from what I can see in the logging.
Configuration from “dovecot -n”:
auth_debug = yes
auth_mechanisms = plain login
auth_verbose = yes
auth_worker_max_count = 5
base_dir = /var/run/dovecot
debug_log_path = /var/log/dovecot/debug.log
default_internal_group = ardaemail
default_internal_user = ardaemail
dict {
quota = mysql:/etc/dovecot/inc.d/inc.quota.user.sql.conf
}
info_log_path = /var/log/dovecot/info.log
instance_name = ec2-us-east-1a-arda-mail-001.ardaemail.com
listen = 172.16.0.200
log_path = /var/log/dovecot/general.log
login_greeting = Welcome to ArdaEmail
mail_debug = yes
mail_gid = ardaemail
mail_location = maildir:/mnt/mail/%d/%n
mail_plugins = " quota"
mail_uid = ardaemail
maildir_stat_dirs = yes
namespace inbox {
inbox = yes
location =
mailbox Archive {
auto = subscribe
special_use = \Archive
}
mailbox Drafts {
auto = no
special_use = \Drafts
}
mailbox Junk {
auto = no
autoexpunge = 30 days
special_use = \Junk
}
mailbox "Junk E-mail" {
auto = no
autoexpunge = 30 days
special_use = \Junk
}
mailbox "Junk Email" {
auto = no
autoexpunge = 30 days
special_use = \Junk
}
mailbox Sent {
auto = no
special_use = \Sent
}
mailbox "Sent Items" {
auto = no
special_use = \Sent
}
mailbox "Sent Messages" {
auto = no
special_use = \Sent
}
mailbox Spam {
auto = subscribe
autoexpunge = 30 days
special_use = \Junk
}
mailbox Trash {
auto = no
autoexpunge = 30 days
special_use = \Trash
}
prefix =
}
passdb {
args = /etc/dovecot/inc.d/inc.sql.conf
driver = sql
}
plugin {
imapsieve_mailbox1_before = file:/etc/dovecot/sieve.d/spam/learn-spam.sieve
imapsieve_mailbox1_causes = COPY
imapsieve_mailbox1_name = Junk
imapsieve_mailbox2_before = file:/etc/dovecot/sieve.d/spam/learn-ham.sieve
imapsieve_mailbox2_causes = COPY
imapsieve_mailbox2_from = Junk
imapsieve_mailbox2_name = *
quota = dict:quota::proxy::quota
quota_exceeded_message = 452 4.2.2 Mailbox is full and cannot receive any more emails
recipient_delimiter = +
sieve_after = /etc/dovecot/sieve.d/after/
sieve_before = /etc/dovecot/sieve.d/before/
sieve_extensions = +spamtest +spamtestplus +virustest +notify +imapflags
sieve_global_extensions = +vnd.dovecot.pipe
sieve_pipe_bin_dir = /etc/dovecot/sieve
sieve_plugins = sieve_imapsieve sieve_extprograms
sieve_quota_max_storage = 0
sieve_spamtest_max_header = X-Spam-Score: -?[[:digit:]]+\.[[:digit:]]+ / (-?[[:digit:]]+\.[[:digit:]])
sieve_spamtest_status_header = X-Spam-Score: (-?[[:digit:]]+\.[[:digit:]]+) / -?[[:digit:]]+\.[[:digit:]]
sieve_spamtest_status_type = score
}
protocols = imap lmtp
service auth-worker {
group = ardaemail
user = ardaemail
}
service auth {
group = ardaemail
unix_listener /var/spool/postfix/private/auth {
group = postfix
mode = 0666
user = postfix
}
unix_listener auth-userdb {
group = ardaemail
mode = 0666
user = ardaemail
}
user = ardaemail
}
service imap-login {
inet_listener imap {
port = 0
}
inet_listener imaps {
port = 993
ssl = yes
}
}
service imap {
group = ardaemail
user = ardaemail
}
service lmtp {
unix_listener /var/spool/postfix/private/dovecot-lmtp {
group = postfix
mode = 0666
user = postfix
}
}
service pop3-login {
inet_listener pop3 {
port = 0
}
inet_listener pop3s {
port = 0
}
}
service submission-login {
inet_listener submission {
port = 0
}
}
ssl = required
ssl_cert = </etc/letsencrypt/live/mail.ardaemail.com/fullchain.pem
ssl_client_ca_dir = /etc/ssl/certs
ssl_dh = # hidden, use -P to show it
ssl_key = # hidden, use -P to show it
ssl_prefer_server_ciphers = yes
userdb {
args = /etc/dovecot/inc.d/inc.sql.conf
driver = sql
}
verbose_ssl = yes
protocol imap {
mail_plugins = " quota imap_sieve imap_quota quota"
}
protocol lmtp {
mail_plugins = " quota sieve imap_quota quota"
}
Configuration for “/etc/dovecot/inc.d/inc.quota.user.sql.conf”:
connect = host=mydb dbname=mydb user=mydbuser password=mydbpass
map {
pattern = priv/quota/storage
table = mailbox
username_field = email
value_field = size
}
map {
pattern = priv/quota/limit/storage
table = mailbox
username_field = email
value_field = quota
}
map {
pattern = priv/quota/messages
table = mailbox
username_field = email
value_field = size_messages
}
map {
pattern = priv/quota/limit/storage
table = mailbox
username_field = email
value_field = quota_messages
}
Configuration from “/etc/dovecot/inc.d/inc.sql.conf”:
driver = mysql
connect = host=mydb dbname=mydb user=mydbuser password=mydbpass
default_pass_scheme = BLF-CRYPT
password_query = SELECT email AS user, password FROM mailbox WHERE email = '%u';
user_query = SELECT maildir AS home, 10000 AS uid, 10000 AS gid, 'dict:quota::proxy::quota' AS quota FROM mailbox WHERE email = '%u';
iterate_query = SELECT email AS username, size, 'dict:quota::proxy::quota' AS quota FROM mailbox ORDER BY domainName ASC, email ASC;
Database Table: “mailbox”
CREATE TABLE mailbox (
id int(11) unsigned NOT NULL AUTO_INCREMENT,
uuid varchar(36) NOT NULL,
domainUuid varchar(36) NOT NULL,
domainName varchar(96) DEFAULT NULL,
username varchar(96) NOT NULL DEFAULT '',
password varchar(128) NOT NULL,
firstname varchar(32) NOT NULL,
lastname varchar(32) NOT NULL,
email varchar(192) NOT NULL DEFAULT '',
maildir text NOT NULL,
quota bigint(24) NOT NULL DEFAULT 0,
quota_messages bigint(24) NOT NULL DEFAULT 0,
size bigint(24) NOT NULL DEFAULT 0,
size_messages bigint(24) NOT NULL DEFAULT 0,
isActive tinyint(1) NOT NULL DEFAULT 1,
createdAt datetime NOT NULL DEFAULT current_timestamp(),
updatedAt datetime NOT NULL DEFAULT current_timestamp() ON UPDATE current_timestamp(),
PRIMARY KEY (id),
UNIQUE KEY UNIQUE_UUID (uuid)
) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_general_ci;
Where “username” is the first part of the email address and “email” is the full email address. I need to query using the full email address because of possible cross domain email addresses.
Output from “doveadm quota get -A”:
Username Quota name Type Value Limit %
HIDDEN quota STORAGE 570742 - 0
HIDDEN quota MESSAGE 0 - 0
HIDDEN quota STORAGE 1 - 0
HIDDEN quota MESSAGE 0 - 0
HIDDEN quota STORAGE 1 - 0
HIDDEN quota MESSAGE 0 - 0
HIDDEN quota STORAGE 1068 - 0
HIDDEN quota MESSAGE 0 - 0
HIDDEN quota STORAGE 462 - 0
HIDDEN quota MESSAGE 0 - 0
HIDDEN quota STORAGE 41165 - 0
HIDDEN quota MESSAGE 0 - 0
HIDDEN quota STORAGE 1598246 - 0
HIDDEN quota MESSAGE 0 - 0
HIDDEN quota STORAGE 695776 - 0
HIDDEN quota MESSAGE 0 - 0
I have no idea what is wrong and I have been at this for over 12 hours trying to get the quotas to work. What I am expecting us when emails are delivered for the “quota” and “messages” columns in the “mailbox” table to be updated and when I run “doveadm quota get -A” to show me the current usages and limits for each mailbox. Once I have that working, I will move on to actually rejecting emails via Postfix + Dovecot LMTP when a users mailbox is full but right now, I am just trying to get this to work.
Any help is greatly appreciated. Also, I tried creating an account for the mailing list but never received the email.
Thank you!
On 30/05/2024 02:21 EEST Adam Miller via dovecot <dovecot@dovecot.org> wrote:
Ubuntu Version: Linux 6.2.0-1017-aws x86_64 Ubuntu 22.04.4 LTS nfs4
Postfix: Version: 3.6.4
Dovecot Version: 2.3.16 (7e2e900c1a)
Pigeonhole Version: 0.5.16 (09c29328)
Rspamd Version: 3.8.4
Protocols: IMAP, LMTP, SMTP
Setup: I have an email server running with virtual domains and virtual mailboxes that is currently hosting about 10GB of email across 15 domains and 8 mailboxes which will expand drastically once I get this to work. The server is an AWS EC2 instance with an AWS RDS instance for the database and using AWS EFS for storage. Everything works quite well except quotas!
Problem: I am having an issue with getting quotas to work. When using “doveadm”, I can list the mailboxes however the “Limit” column is just a hyphen “-“ like there is no limit even though every account is set to 10GB for their limit in the database. At the same time, the actual used storage and messages is not getting updated in the database either yet there are no errors from what I can see in the logging.
For new installations, we suggest using
quota = count quota_vsizes = yes
and using quota_clone plugin to store quota information in SQL database.
Perhaps this works better for you?
For limits, I cannot see any quota_rule's in your config, and there is no
doveadm user someusername
output in your email.
Aki
Thank you for the reply. I was actually able to get it all working however now I am having an issue trying to get the quota warning emails to work. Not sure what the deal is but once I have hammered on it enough and it still isn't working, I will reply to ask for additional help.
As for what the issue was, I am not entirely positive. I started over with the quota configs and slowly added introduced new configs and it started working.
Thank you!
On 5/29/24, 21:55, "Aki Tuomi via dovecot" <dovecot@dovecot.org <mailto:dovecot@dovecot.org>> wrote:
On 30/05/2024 02:21 EEST Adam Miller via dovecot <dovecot@dovecot.org <mailto:dovecot@dovecot.org>> wrote:
Ubuntu Version: Linux 6.2.0-1017-aws x86_64 Ubuntu 22.04.4 LTS nfs4
Postfix: Version: 3.6.4
Dovecot Version: 2.3.16 (7e2e900c1a)
Pigeonhole Version: 0.5.16 (09c29328)
Rspamd Version: 3.8.4
Protocols: IMAP, LMTP, SMTP
Setup: I have an email server running with virtual domains and virtual mailboxes that is currently hosting about 10GB of email across 15 domains and 8 mailboxes which will expand drastically once I get this to work. The server is an AWS EC2 instance with an AWS RDS instance for the database and using AWS EFS for storage. Everything works quite well except quotas!
Problem: I am having an issue with getting quotas to work. When using “doveadm”, I can list the mailboxes however the “Limit” column is just a hyphen “-“ like there is no limit even though every account is set to 10GB for their limit in the database. At the same time, the actual used storage and messages is not getting updated in the database either yet there are no errors from what I can see in the logging.
For new installations, we suggest using
quota = count quota_vsizes = yes
and using quota_clone plugin to store quota information in SQL database.
Perhaps this works better for you?
For limits, I cannot see any quota_rule's in your config, and there is no
doveadm user someusername
output in your email.
Aki
dovecot mailing list -- dovecot@dovecot.org <mailto:dovecot@dovecot.org> To unsubscribe send an email to dovecot-leave@dovecot.org <mailto:dovecot-leave@dovecot.org>
On 30/05/2024 20:06, Adam Miller via dovecot wrote:
however now I am having an issue trying to get the quota warning emails to work.
Your original post did not show a "service quota-warning" section where you tell dovecot what to run, I suggest you fully read everything to do with quota on the wiki (the relevant wiki files are also included in source packages)
-- Regards, Noel Butler
Thank you!
At the time, I was trying to get the most basic of quotas working which I have now successfully accomplished! I am happy report that I also have the warning emails working.
Next, my plan is to figure out a way to write custom HTML warning emails using the MJML templating engine. My thought was to have a bash script kick off a Python script which would grab the HTML email using an MJML template stored in a database and then pass back the HTML email it to the bash script to finally send it out. I know this might sound confusing and why would anyone want to do this, but my end goal is to build an email hosting service so having everything branded, secure and functioning well is at the top of the list. Is it possible that instead of a bash script for the warning emails to use a Python script instead?
I also must investigate load balancing or at the very least, determining the best approach to scalability and high availability.
Thank you!
On 6/1/24, 03:18, "Noel Butler via dovecot" <dovecot@dovecot.org <mailto:dovecot@dovecot.org>> wrote:
On 30/05/2024 20:06, Adam Miller via dovecot wrote:
however now I am having an issue trying to get the quota warning emails to work.
Your original post did not show a "service quota-warning" section where you tell dovecot what to run, I suggest you fully read everything to do with quota on the wiki (the relevant wiki files are also included in source packages)
-- Regards, Noel Butler
dovecot mailing list -- dovecot@dovecot.org <mailto:dovecot@dovecot.org> To unsubscribe send an email to dovecot-leave@dovecot.org <mailto:dovecot-leave@dovecot.org>
On 01/06/2024 20:23, Adam Miller via dovecot wrote:
Thank you!
At the time, I was trying to get the most basic of quotas working which I have now successfully accomplished! I am happy report that I also have the warning emails working.
Excellent.
Is it possible that instead of a bash script for the warning emails to use a Python script instead?
Never been a fan of python, too much of a resource hog, even compared to perl, but as long as the variables are interpreted correctly, yes it should work.
I also must investigate load balancing or at the very least, determining the best approach to scalability and high availability.
We've used NFS for years without problems, never used dovecot's director service either, however we use hardware load balancers, done right, this is simplest and most robust method, add/delete/down-for-update front end servers at your will without affecting anything, as for backend, don't use junk, I've found EMC storage gear very reliable, but know that NetApp is too.
Over the years I've read about and witnessed many businesses with multi-day outages using clustered file systems that take out everything when they have a hissy fit, so I avoid them at all cost. NFS might be simplicity, but that means far fewer things to go wrong, and why some very large well known mail providers use it too.
-- Regards, Noel Butler
participants (3)
-
Adam Miller
-
Aki Tuomi
-
Noel Butler