[Dovecot] Load spikes on NFS server, multiple index updaters.
We are occasionally experiencing trouble where the NFS server's load will shoot over 60+. (Normal of sub 1.0).
I have been hunting this for a while, and I believe it comes down to "deliver".
System setup:
NFS servers: x4540 Solaris 10 x64 ZFS over NFS. NFS clients: Solaris 10 x64 postfix-2.4.1 with dovecot-1.1.11 deliver.
What appears to happen, when I check for nfsstat per process, is that I see 4 processes (in this case on vmx04) taking up majority of NFS ops:
root@vmx04:/var/tmp# ./nfsclientstats.pl process read write readdir getattr setattr lookup access create remove rename mkdir rmdir 24303 0 0 1 19 0 190 171 0 0 0 0 0 24551 0 0 1 18 0 180 162 0 0 0 0 0 26099 0 0 1 18 0 180 162 0 0 0 0 0 295 0 0 1 18 0 180 162 0 0 0 0 0 6793 3 0 0 0 0 5 5 0 0 0 0 0 7234 0 1 0 2 0 9 9 0 1 0 0 0
Checking what these processes are doing, I find the following happening:
26099: getdents64(8, 0xCE7A4000, 8192) = 8136 26099: stat64("/export/censored/mail/cur/1223013930.V4700010I69f93eM483098.vmx02.unix:2,S", 0x08047930) = 0 26099: stat64("/export/censored/mail/cur/1222066290.V4700007I67562bM241839.vmx04.unix:2,S", 0x08047930) = 0 26099: stat64("/export/censored/mail/cur/1225325373.V4700008I94a1f9M286935.vmx04.unix:2,S", 0x08047930) = 0 26099: stat64("/export/censored/mail/cur/1236170307.V4700002I67ca03M310418.vmx06.unix:2,", 0x08047930) = 0 26099: stat64("/export/censored/mail/cur/1223581462.V4700011I69ffd6M720814.vmx02.unix:2,S", 0x08047930) = 0
Very well, so it is rebuilding the dovecot.index, or recalculating the user's quota usage.
Is the directory large?
root@vmx04# ls -l /export/censored/mail/cur/|wc -l 199626
You bet! But what is annoying is that if I also check process 24303, 24551 and 295, they are scanning the SAME user's directory.
295: stat64("/export/censored/mail/cur/1230544947.V4700004I11d1d7fM492433.vmx04.unix:2,S", 0x08047930) = 0 295: stat64("/export/censored/mail/cur/1223003964.V4700007I68932dM763546.vmx04.unix:2,S", 0x08047930) = 0
So, in vmx04 we have 4 processes working in one user's giant directory, and on the other vmx clients, many more.
Could the semantics to 're-computing dovecot.index' be done such that the first "deliver" process locks it to do the work, and sub-sequent deliver processes will return temporary failures, until the work has finished.
Has it been already addresses in dovecot-1.1.18?
Advice please.
Lund
-- Jorgen Lundman | <lundman@lundman.net> Unix Administrator | +81 (0)3 -5456-2687 ext 1017 (work) Shibuya-ku, Tokyo | +81 (0)90-5578-8500 (cell) Japan | +81 (0)3 -3375-1767 (home)
On Aug 25, 2009, at 10:06 PM, Jorgen Lundman wrote:
26099: stat64("/export/censored/mail/cur/ 1223013930.V4700010I69f93eM483098.vmx02.unix:2,S", 0x08047930) = 0
Are these old files, or why don't they contain the ,S=1234 in
filename? That would help a lot when recaculating Maildir++ quota.
But Maildir++ quota is supposed to work without locks.. :)
Do you need Maildir++ quota at all? With v1.2 you could use dict quota
with file backend. It'll use dovecot.index.cache when recalculating
quota, although it doesn't do that unless the quota is lost for some
reason (so about never).
Timo Sirainen wrote:
Are these old files, or why don't they contain the ,S=1234 in filename? That would help a lot when recaculating Maildir++ quota.
Ah legacy reasons :) We used to use UFS (with quotas) inside zvol on the zfs server. But this panics, and has performance issues.
We changed to plain zfs (without quotas) and enabled soft-quotas/Maildir++. So yes, some mails are so old that they do not contain ,S=1234. All new mail does:
-rw------- 1 5091 mail 1676 Aug 26 12:31 1251257478.M554367P15043.vmx05.unix,S=1676,W=1711 -rw------- 1 5091 mail 1674 Aug 26 12:33 1251257579.M948723P22818.vmx04.unix,S=1674,W=1709
But Maildir++ quota is supposed to work without locks.. :)
Sure, but 20 processes all scanning the same directory, over 12 servers, is unpleasant. :)
We only needed a temporary solution. Long term, we have already tested ZFS's user-quota, and this is where we will go. Getting enough momentum to get the process started (management...) is what has been frustrating. Would you say this trouble is soft-quotas scanning, and not just generic index rebuild?
The reason I ask is that; if it is related to quotas, that gives me yet-another reason to move to zfs quotas (and the trouble will go away). If it is normal index rebuilding, it will still happen even after using ZFS quotas.
Thanks for your reply though,
Lund
-- Jorgen Lundman | <lundman@lundman.net> Unix Administrator | +81 (0)3 -5456-2687 ext 1017 (work) Shibuya-ku, Tokyo | +81 (0)90-5578-8500 (cell) Japan | +81 (0)3 -3375-1767 (home)
On Aug 25, 2009, at 11:39 PM, Jorgen Lundman wrote:
The problem is Maildir++ quota implementation. I did it exactly as
Courier's specs said it should be done.. It doesn't use Dovecot's
indexes at all, so I could get rid of this problem by changing it to
use indexes. But that'll complicate the code and makes it more or less
what the dict quota with file backend does already..
Yeah, zfs quotas gets rid of this problem. Maybe I should also start
recommending people to use dict quota with v1.2.. Hmm. Although I
think there was also some rare issue with it that I should fix.
Ah great. I was just about to move the customer's directory to the test servers and try with, and without, quotas to prove it. This saves me the hassle.
We also have customers with one set quota for many mail accounts (under many domains, from LDAP, same UID) which is why file-system quotas suit us. But possibly dict-quotas can also handle this, I don't know. (We need the quota for CGI services, etc.. anyway).
Thanks for your assistance,
Lund
-- Jorgen Lundman | <lundman@lundman.net> Unix Administrator | +81 (0)3 -5456-2687 ext 1017 (work) Shibuya-ku, Tokyo | +81 (0)90-5578-8500 (cell) Japan | +81 (0)3 -3375-1767 (home)
participants (2)
-
Jorgen Lundman
-
Timo Sirainen