[Dovecot] compressed mboxes very slow
Stan Hoeppner
stan at hardwarefreak.com
Tue May 10 08:06:54 EEST 2011
On 5/9/2011 6:20 PM, Kamil Jońca wrote:
> 5. connect with mutt and select one of gzipped folders
> 6. connect with fetchmail and select another gzipped folder (with the
> same contents as in 5)
Does the problem occur with only one client, or are two or more clients
required to reproduce the problem? If the former, you're simply
unnecessarily confusing things by mentioning two clients in the problem
reproduction case here.
> grepping strace logs shows that in both cases mboxes are reread
> regularly :( moreover there's no inotify_init (...) call by any dovecot
> process.
> Any ideas?
The first time the mbox file is read its filesystem blocks will be
cached. The actual file read will take 1 second max on 10 year old
hardware, 0.2 seconds on modern hardware. Thus, each subsequent time
Dovecot 're-reads' the file, it will occur at RAM speed, something
between 3.2 and 12.8GB/s depending on system age.
Thus, if it's taking 10 minutes to build the indexes for this gzipped
mbox file, the cause of that elapsed 10 minutes isn't going to be the
re-reading of the file, as this operation will be completing in
microseconds.
If you go back to 1.2.15 and strace Dovecot, do you see the same
re-reading of the mbox files after each 20 or so messages? This
behavior may be normal. I'm not familiar with the code or I'd have told
you if this is the case. Regardless, I doubt re-reading is the cause of
the slowness. There's got to be something else in the loop besides the
file read eating up the CPU time. Given it's 10 minutes for a 20MB file
it would seem logical that a large wait is being inserted or Dovecot is
waiting for a lock to expire--something along these lines. I'd doubt
Dovecot is actually doing anything most of this 10 minutes.
I think you need to find out what Dovecot 2.x is waiting on that 1.2.15
wasn't.
--
Stan
More information about the dovecot
mailing list