[Dovecot] Maybe wrong quota recalculation behaviour on overquota delivery try
Hi all, we are experencing some, may be strange, slowness in our dovecot/postfix mail system. We have three dovecot 2.0.1 server with LMTP, POP3 and IMAP enabled. Three shared OCFS2 filesystem are used for mail storage. We have too a load balancer based on dovecot director in front of these three server. Our SMTP cluster receives mail and send by LMTP to load balancer.
Everything work flowless, but sometime, under heavy load, three server slow down a lot. Our mail gateway logs a lot of: status=deferred (host 10.194.1.20[10.194.1.20] said: 451 4.4.0 Remote server not answering (DATA output timeout)
IP 10.194.1.20 is dovecot director.
Looking at mbox servers log nothing seem to happen, but stracing lmtp process we can see stat system calls to all Maildir files, as if dovecot was recalculating quota and regenerating maildirsize file.
It happens always when LMTP tries to deliver a mail to an overquota account.
Is this possible and right? How can we look deeper in system to have more information?
Best regards
Luca
PS: dovecot configuration for maildir servers is:
# 2.0.1: /etc/dovecot/dovecot.conf # OS: Linux 2.6.26-2-686-bigmem i686 Debian 5.0.5 auth_master_user_separator = * auth_verbose = yes disable_plaintext_auth = no mail_location = maildir:~/Maildir mail_plugins = $mail_plugins notify mail_log quota managesieve_notify_capability = mailto managesieve_sieve_capability = comparator-i;octet comparator-i;ascii-casemap fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date spamtest spamtestplus virustest passdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql } plugin { mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename mail_log_fields = uid box msgid from size quota = maildir:User quota quota_warning = storage=95%% quota-warning.sh 95 %u quota_warning2 = storage=80%% quota-warning.sh 80 %u sieve = ~/.dovecot.sieve sieve_dir = ~/sieve sieve_global_dir = /etc/dovecot/sieve/global/ sieve_global_path = /etc/dovecot/sieve/default.sieve } protocols = imap pop3 lmtp sieve service auth { unix_listener auth-userdb { group = dovecot mode = 0600 user = dovecot } } service director { inet_listener { port = 8100 } } service lmtp { inet_listener lmtp { address = 10.194.1.31 127.0.0.1 ::1 port = 24 } } service managesieve-login { inet_listener sieve { address = 10.194.1.31 127.0.0.1 ::1 port = 4190 } process_min_avail = 0 service_count = 1 } service quota-warning { executable = script /virtual/shared/quota-warning.sh unix_listener quota-warning { user = vmail } user = dovecot } ssl_cert =
On Thu, 2010-09-02 at 19:14 +0200, Luca Palazzo wrote:
Looking at mbox servers log nothing seem to happen, but stracing lmtp process we can see stat system calls to all Maildir files, as if dovecot was recalculating quota and regenerating maildirsize file.
It happens always when LMTP tries to deliver a mail to an overquota account.
Yes, that's pretty much how Maildir++ quota works. If maildirsize file doesn't only contain a single line showing the quota, or if it's older than 15 minutes, it's recalculated. The maildir filenames really should have ,S=size in the filenames, otherwise quota recalculation is horribly slow.
Maybe you should switch to dict quota with file backend. http://wiki2.dovecot.org/Quota/Dict
Hi Timo, thanks you for reply. I'm pretty sure you are right. Quota recalculation based on Maildir++ standard is too I/O intensive for our system because we have lot of maildirs migrated from old system. Mail files are missing often size in filename so now I understand why so much stat calls. Is there any script/util in dovecot to fix filenames?
Thank you.
Luca
On 09/02/2010 07:32 PM, Timo Sirainen wrote:
On Thu, 2010-09-02 at 19:14 +0200, Luca Palazzo wrote:
Looking at mbox servers log nothing seem to happen, but stracing lmtp process we can see stat system calls to all Maildir files, as if dovecot was recalculating quota and regenerating maildirsize file.
It happens always when LMTP tries to deliver a mail to an overquota account.
Yes, that's pretty much how Maildir++ quota works. If maildirsize file doesn't only contain a single line showing the quota, or if it's older than 15 minutes, it's recalculated. The maildir filenames really should have ,S=size in the filenames, otherwise quota recalculation is horribly slow.
Maybe you should switch to dict quota with file backend. http://wiki2.dovecot.org/Quota/Dict
On Fri, 2010-09-03 at 08:19 +0200, Luca Palazzo wrote:
Hi Timo, thanks you for reply. I'm pretty sure you are right. Quota recalculation based on Maildir++ standard is too I/O intensive for our system because we have lot of maildirs migrated from old system. Mail files are missing often size in filename so now I understand why so much stat calls. Is there any script/util in dovecot to fix filenames?
The problem with changing filenames is that it changes also the maildir base filename. So that requires changing also dovecot-uidlist. But it gets worse if you use base filenames as POP3 UIDLs, then a change in it causes POP3 clients to redownload those mails.
participants (2)
-
Luca Palazzo
-
Timo Sirainen