[Dovecot] lstats on all files in Maildir when opening an e-mail.
Hi,
I'm running RC19 on a Solaris 10 SPARC box.
It seems that whenever I open an e-mail in any folder via IMAP, dovecot appears to scan every other folder in the Maildir.
A truss reveals that it is lstat-ing every e-mail in every directory.
I'm sorry I am sending only a fragment of the truss, it's really too long... but is this behaviour expected?
Cheers, Tan Shao Yi
lstat64("/var/qmail/maildirs/AA/AA/test@pacific.net.sg/Maildir/.qmail/new/1170337156.M209211P18431V025055F0I0000DAF0_0.recess1,S=2596", 0xFFBFF040) = 0 lstat64("/var/qmail/maildirs/AA/AA/test@pacific.net.sg/Maildir/.qmail/new/1170337594.M159985P18549V025055F0I0001891B_0.recess1,S=4111", 0xFFBFF040) = 0 lstat64("/var/qmail/maildirs/AA/AA/test@pacific.net.sg/Maildir/.qmail/new/1170338379.M564254P18933V025055F0I000196A3_0.recess1,S=3630", 0xFFBFF040) = 0 lstat64("/var/qmail/maildirs/AA/AA/test@pacific.net.sg/Maildir/.qmail/new/1170344035.M859029P24106V025055F0I000198E4_0.recess1,S=4254", 0xFFBFF040) = 0 lstat64("/var/qmail/maildirs/AA/AA/test@pacific.net.sg/Maildir/.qmail/new/1170344816.M69242P24829V025055F0I000199CB_0.recess1,S=3810", 0xFFBFF040) = 0 lstat64("/var/qmail/maildirs/AA/AA/test@pacific.net.sg/Maildir/.qmail/new/1170347789.M408186P27905V025055F0I00019A8F_0.recess1,S=3922", 0xFFBFF040) = 0 lstat64("/var/qmail/maildirs/AA/AA/test@pacific.net.sg/Maildir/.qmail/new/1170378895.M589722P25748V025055F0I00019279_0.recess1,S=2782", 0xFFBFF040) = 0 lstat64("/var/qmail/maildirs/AA/AA/test@pacific.net.sg/Maildir/.qmail/new/1170380726.M524353P29456V025055F0I0001AEE1_0.recess1,S=3619", 0xFFBFF040) = 0 lstat64("/var/qmail/maildirs/AA/AA/test@pacific.net.sg/Maildir/.qmail/new/1170381420.M702545P517V025055F0I0001B208_0.recess1,S=3028", 0xFFBFF040) = 0 getdents64(8, 0xFEE72000, 8192) = 0 close(8) = 0 lstat64("/var/qmail/maildirs/AA/AA/test@pacific.net.sg/Maildir/.qmail/maildirfolder", 0xFFBFF148) = 0 lstat64("/var/qmail/maildirs/AA/AA/test@pacific.net.sg/Maildir/.qmail/cur", 0xFFBFF148) = 0 openat(-3041965, "/var/qmail/maildirs/AA/AA/test@pacific.net.sg/Maildir/.qmail/cur", O_RDONLY|O_NDELAY|O_LARGEFILE) = 8 fcntl(8, F_SETFD, 0x00000001) = 0 fstat64(8, 0xFFBFEEE8) = 0 getdents64(8, 0xFEE72000, 8192) = 1088 lstat64("/var/qmail/maildirs/AA/AA/test@pacific.net.sg/Maildir/.qmail/cur/1169092926.M656431P6118V04010002I000023E9_0.recess1,S=4701:2,S", 0xFFBFF040) = 0 lstat64("/var/qmail/maildirs/AA/AA/test@pacific.net.sg/Maildir/.qmail/cur/1168983168.M76366P8906V04010002I000027AB_0.recess1,S=2345:2,S", 0xFFBFF040) = 0
On Fri, 2007-02-02 at 18:35 +0800, Tan Shao Yi wrote:
Hi,
I'm running RC19 on a Solaris 10 SPARC box.
It seems that whenever I open an e-mail in any folder via IMAP, dovecot appears to scan every other folder in the Maildir.
A truss reveals that it is lstat-ing every e-mail in every directory.
I'm sorry I am sending only a fragment of the truss, it's really too long... but is this behaviour expected?
Are you using dirsize quota backend? That's the only reason I can think of for this. dirsize backend really sucks with maildir.
Tan Shao Yi wrote:
On Fri, 2 Feb 2007, Timo Sirainen wrote:
Are you using dirsize quota backend? That's the only reason I can think of for this. dirsize backend really sucks with maildir.
Hi Timo,
Thanks! You hit it spot-on.
Thanks for the tip-off.
Cheers.
Hi guys - So what can we use instead? We've just had 2 days downtime because Dovecot was using 100% of the disk. As reported above - every action on a user mailbox seems to lstat every file in the user's home.
So we've disabled quota completely and these millions of lstats have stopped.
We're running virtual users (all under a vmail system user) so fs quota is not workable. What do you recommend for a Maildir?
This is bad enough to almost work having Dovecot throw a: WARNING: dirsize not recommended with Maildir format.
Best wishes, Daniel
-- Squirrelmail Stable 1.4.8 PHP 5.x Hardened with Eaccelerator Apache 2.x Imapproxy over Dovecot/Maildir all running on Gentoo Linux for ~5,000 users.
On Tue, 27 Feb 2007, Daniel Watts wrote:
Hi guys - So what can we use instead? We've just had 2 days downtime because Dovecot was using 100% of the disk. As reported above - every action on a user mailbox seems to lstat every file in the user's home.
So we've disabled quota completely and these millions of lstats have stopped.
We're running virtual users (all under a vmail system user) so fs quota is not workable. What do you recommend for a Maildir?
This is bad enough to almost work having Dovecot throw a: WARNING: dirsize not recommended with Maildir format.
Hi Daniel,
I am now using Maildir++ quotas (http://wiki.dovecot.org/Quota/Maildir).
Cheers.
Hi Tan Shao,
Thanks for the fast reply =) My sysadmin also found that link and has tried putting it into place. Seems like it has solved the problem nicely.
We are still seeing 99%-100% disk usage though which is a little worrying:
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s
avgrq-sz avgqu-sz await svctm %util
sda 4.80 16.50 86.50 34.70 3464.80 730.60 1732.40
365.30 34.62 27.38 224.38 8.25 100.04
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s
avgrq-sz avgqu-sz await svctm %util
sda 4.00 8.99 74.93 33.57 2618.98 1175.72 1309.49
587.86 34.98 20.10 187.70 9.21 99.94
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s
avgrq-sz avgqu-sz await svctm %util
sda 1.80 36.84 49.25 47.75 1508.71 1114.61 754.35
557.31 27.05 41.22 424.25 10.32 100.10
Do you get think kind of load when Dovecot is busy? We have about 50 dovecot processes running.
Best wishes, Daniel
Tan Shao Yi wrote:
On Tue, 27 Feb 2007, Daniel Watts wrote:
Hi guys - So what can we use instead? We've just had 2 days downtime because Dovecot was using 100% of the disk. As reported above - every action on a user mailbox seems to lstat every file in the user's home.
So we've disabled quota completely and these millions of lstats have stopped.
We're running virtual users (all under a vmail system user) so fs quota is not workable. What do you recommend for a Maildir?
This is bad enough to almost work having Dovecot throw a: WARNING: dirsize not recommended with Maildir format.
Hi Daniel,
I am now using Maildir++ quotas (http://wiki.dovecot.org/Quota/Maildir).
Cheers.
Hi Daniel,
Did you just implement Maildir++ quotas? If you did, it may be busy recalculating the "maildirsize" file for the first time.
There's an interesting read on how/when this is done: http://www.inter7.com/courierimap/README.maildirquota.html
Otherwise you may need to trace what the processes are really doing on disk.
I'm sorry I can't share with you our system load; we are running a department server with about 30-odd users (and it's on a SAN!)... so I don't think the load that we have is really that significant. :P
Cheers, TAN Shao Yi
On Tue, 27 Feb 2007, Daniel Watts wrote:
Hi Tan Shao,
Thanks for the fast reply =) My sysadmin also found that link and has tried putting it into place. Seems like it has solved the problem nicely.
We are still seeing 99%-100% disk usage though which is a little worrying:
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util sda 4.80 16.50 86.50 34.70 3464.80 730.60 1732.40 365.30 34.62 27.38 224.38 8.25 100.04
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util sda 4.00 8.99 74.93 33.57 2618.98 1175.72 1309.49 587.86 34.98 20.10 187.70 9.21 99.94
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util sda 1.80 36.84 49.25 47.75 1508.71 1114.61 754.35 557.31 27.05 41.22 424.25 10.32 100.10
Do you get think kind of load when Dovecot is busy? We have about 50 dovecot processes running.
Best wishes, Daniel
Tan Shao Yi wrote:
On Tue, 27 Feb 2007, Daniel Watts wrote:
Hi guys - So what can we use instead? We've just had 2 days downtime because Dovecot was using 100% of the disk. As reported above - every action on a user mailbox seems to lstat every file in the user's home.
So we've disabled quota completely and these millions of lstats have stopped.
We're running virtual users (all under a vmail system user) so fs quota is not workable. What do you recommend for a Maildir?
This is bad enough to almost work having Dovecot throw a: WARNING: dirsize not recommended with Maildir format.
Hi Daniel,
I am now using Maildir++ quotas (http://wiki.dovecot.org/Quota/Maildir).
Cheers.
participants (3)
-
Daniel Watts
-
Tan Shao Yi
-
Timo Sirainen