[Dovecot] best way to activate quota
Hello,
I'm planning to activate quota control in dovecot, with maildir quota backend. I have about 70k users in my system directed to 4 backend servers (with a director to ensure that a user is always directed to the same server).
I have tried to activate it in one of my nodes. The problem is that load of it has increase a lot, as much as the system was unusable (maildir is in nfs storage, with indexes in local disks).
What do you think is the best to activate it?
I have think that I could use doveadm commands to calculate an initial quota (on low load periods) and then activate quota, hoping that although quota is not correct, it would be used. Should this work?
Am 07.09.2012 12:11, schrieb Angel L. Mateo:
Hello,
I'm planning to activate quota control in dovecot, with maildirquota backend. I have about 70k users in my system directed to 4 backend servers (with a director to ensure that a user is always directed to the same server).
sounds good, first look
I have tried to activate it in one of my nodes. The problem is thatload of it has increase a lot, as much as the system was unusable (maildir is in nfs storage, with indexes in local disks).
that might not be the optimal storage solution
What do you think is the best to activate it?
yes ,until its better investigated, cause your systems shouldnt get unusable
I have think that I could use doveadm commands to calculate aninitial quota (on low load periods) and then activate quota, hoping that although quota is not correct, it would be used. Should this work?
with 70 k users , you should be able to hire professional help from timo and/or other dovecot specialists however you ever will have to post your config/logs/general-setup-big-picture if asking for help
-- Best Regards MfG Robert Schetterer
El 07/09/12 12:28, Robert Schetterer escribió:
I have tried to activate it in one of my nodes. The problem is thatload of it has increase a lot, as much as the system was unusable (maildir is in nfs storage, with indexes in local disks).
that might not be the optimal storage solution
Why do you say it might be not optimal? I think that nfs storage is a very common scenario.
On 7.9.2012, at 13.11, Angel L. Mateo wrote:
Hello,
I'm planning to activate quota control in dovecot, with maildir quota backend. I have about 70k users in my system directed to 4 backend servers (with a director to ensure that a user is always directed to the same server).
I have tried to activate it in one of my nodes. The problem is that load of it has increase a lot, as much as the system was unusable (maildir is in nfs storage, with indexes in local disks).
What do you think is the best to activate it?
Use dict-file quota instead of Maildir++ quota.
On Tue, Sep 11, 2012 at 07:20:02PM +0300, Timo Sirainen wrote:
On 7.9.2012, at 13.11, Angel L. Mateo wrote:
Hello,
I'm planning to activate quota control in dovecot, with maildir quota backend. I have about 70k users in my system directed to 4 backend servers (with a director to ensure that a user is always directed to the same server).
I have tried to activate it in one of my nodes. The problem is that load of it has increase a lot, as much as the system was unusable (maildir is in nfs storage, with indexes in local disks).
What do you think is the best to activate it?
Use dict-file quota instead of Maildir++ quota.
Sorry to jump into this topic, but can I ask why? Is dict quota is superior over Maildir++ quota in performance in general or it is "only" about the fact, that it's better in the case like above: activating the quota for many users later than the creation of the server itself?
The question is interesting for me as well, as I need to implement quota with maildir (on NFS).
On 12.9.2012, at 1.21, Gábor Lénárt wrote:
I'm planning to activate quota control in dovecot, with maildir quota backend. I have about 70k users in my system directed to 4 backend servers (with a director to ensure that a user is always directed to the same server).
I have tried to activate it in one of my nodes. The problem is that load of it has increase a lot, as much as the system was unusable (maildir is in nfs storage, with indexes in local disks).
What do you think is the best to activate it?
Use dict-file quota instead of Maildir++ quota.
Sorry to jump into this topic, but can I ask why? Is dict quota is superior over Maildir++ quota in performance in general or it is "only" about the fact, that it's better in the case like above: activating the quota for many users later than the creation of the server itself?
Activating quota is equally slow, because it needs to scan sizes of all files. If the size is stored in maildir filename (,S=123) this is pretty fast, otherwise it's pretty slow because Dovecot stat()s the files, but afterwards it stores them to dovecot-uidlist file. So the main difference is that Maildir++ requires rescanning the quota periodically, while dict quota never rescans unless you run doveadm quota rescan.
Il 12/09/2012 00:28, Timo Sirainen ha scritto:
On 12.9.2012, at 1.21, Gábor Lénárt wrote:
Sorry to jump into this topic, but can I ask why? Is dict quota is superior over Maildir++ quota in performance in general or it is "only" about the fact, that it's better in the case like above: activating the quota for many users later than the creation of the server itself? Activating quota is equally slow, because it needs to scan sizes of all files. If the size is stored in maildir filename (,S=123) this is pretty fast, otherwise it's pretty slow because Dovecot stat()s the files, but afterwards it stores them to dovecot-uidlist file. So the main difference is that Maildir++ requires rescanning the quota periodically, while dict quota never rescans unless you run doveadm quota rescan.
But dict quota, for enforce limits, can only be used with dovect as LDA? I'm using Maildir++ with qmail/vpopmail for local delivery and dovecot cannot enforce limits on delivery. With this configuration the only way to update dictionary is "doveadm -A quota rescan".
Is correct?
-- Alessio Cecchi is: @ ILS -> http://www.linux.it/~alessice/ on LinkedIn -> http://www.linkedin.com/in/alessice Assistenza Sistemi GNU/Linux -> http://www.cecchi.biz/ @ PLUG -> ex-Presidente, adesso senatore a vita, http://www.prato.linux.it
El 12/09/12 00:28, Timo Sirainen escribió:
Activating quota is equally slow, because it needs to scan sizes of all files. If the size is stored in maildir filename (,S=123) this is pretty fast, otherwise it's pretty slow because Dovecot stat()s the files, but afterwards it stores them to dovecot-uidlist file. So the main difference is that Maildir++ requires rescanning the quota periodically, while dict quota never rescans unless you run doveadm quota rescan.
doveadm quota rescan (not documented)? or doveadm quota recalc (as documented)? Anyway, I'll use dict instead of maildir backend.
On Wed, Sep 12, 2012 at 01:28:58AM +0300, Timo Sirainen wrote:
Activating quota is equally slow, because it needs to scan sizes of all files. If the size is stored in maildir filename (,S=123) this is pretty fast, otherwise it's pretty slow because Dovecot stat()s the files, but afterwards it stores them to dovecot-uidlist file. So the main difference is that Maildir++ requires rescanning the quota periodically, while dict quota never rescans unless you run doveadm quota rescan.
Ahaa, thanks for your answer.
But ... why is rescan needed sometimes with maildir++? If I don't touch maildir by other softwares ever (I even have maildir_very_dirty_syncs = yes) just dovecot pop3/imap and incoming mails via dovecot's lmtp (no other MDAs, not even the LDA, etc), then why is it needed to rescan periodically with maildir? I don't really understand the difference then which needs quota rescan with maildir++ but not with dict quota during the very same usage environment otherwise.
Anyway I guess, having dict quota with SQL backend is not so a bad idea: I would be able to check incoming mails (at MXs) if target user will have enough space to hold the mail by using the mail size and quota usage (from SQL what dict quota uses as well) on the MXs using eg postfix policy server ... So I won't generate ugly NDRs later, after accepted mails. Not 100% accurate (quota usage may change while mail hits dovecot) but at least it will stop the majority of those quota related NDRs I have problems with currently.
The only thing which keep me away from dict quota with sql backend that it needs much more complex stuff than simply using maildir++ quota, but it's maybe only my lazyness ...
thanks again.
El 12/09/12 13:34, Gábor Lénárt escribió:
On Wed, Sep 12, 2012 at 01:28:58AM +0300, Timo Sirainen wrote:
Activating quota is equally slow, because it needs to scan sizes of all files. If the size is stored in maildir filename (,S=123) this is pretty fast, otherwise it's pretty slow because Dovecot stat()s the files, but afterwards it stores them to dovecot-uidlist file. So the main difference is that Maildir++ requires rescanning the quota periodically, while dict quota never rescans unless you run doveadm quota rescan.
Ahaa, thanks for your answer.
But ... why is rescan needed sometimes with maildir++? If I don't touch maildir by other softwares ever (I even have maildir_very_dirty_syncs = yes) just dovecot pop3/imap and incoming mails via dovecot's lmtp (no other MDAs, not even the LDA, etc), then why is it needed to rescan periodically with maildir? I don't really understand the difference then which needs quota rescan with maildir++ but not with dict quota during the very same usage environment otherwise.
Anyway I guess, having dict quota with SQL backend is not so a bad idea: I would be able to check incoming mails (at MXs) if target user will have enough space to hold the mail by using the mail size and quota usage (from SQL what dict quota uses as well) on the MXs using eg postfix policy server ... So I won't generate ugly NDRs later, after accepted mails. Not 100% accurate (quota usage may change while mail hits dovecot) but at least it will stop the majority of those quota related NDRs I have problems with currently.
The only thing which keep me away from dict quota with sql backend that it needs much more complex stuff than simply using maildir++ quota, but it's maybe only my lazyness ...
You could also use dict quota stored in a file in the user's home directory as maildir++ quota does. This is the way I'm going to use it, not with a sql backend.
-- Angel L. Mateo Martínez Sección de Telemática Área de Tecnologías de la Información y las Comunicaciones Aplicadas (ATICA) http://www.um.es/atica Tfo: 868887590 Fax: 868888337
participants (5)
-
Alessio Cecchi
-
Angel L. Mateo
-
Gábor Lénárt
-
Robert Schetterer
-
Timo Sirainen