[Dovecot] compressed mboxes very slow

[Stan Hoeppner]

On 5/9/2011 6:20 PM, Kamil Jońca wrote:

> 5. connect with mutt and select one of gzipped folders
> 6. connect with fetchmail and select another gzipped folder (with the
> same contents as in 5)

Does the problem occur with only one client, or are two or more clients 
required to reproduce the problem?  If the former, you're simply 
unnecessarily confusing things by mentioning two clients in the problem 
reproduction case here.

> grepping strace logs shows that in both cases mboxes are reread
> regularly :( moreover there's no inotify_init (...) call by any dovecot
> process.

> Any ideas?

The first time the mbox file is read its filesystem blocks will be 
cached.  The actual file read will take 1 second max on 10 year old 
hardware, 0.2 seconds on modern hardware.  Thus, each subsequent time 
Dovecot 're-reads' the file, it will occur at RAM speed, something 
between 3.2 and 12.8GB/s depending on system age.

Thus, if it's taking 10 minutes to build the indexes for this gzipped 
mbox file, the cause of that elapsed 10 minutes isn't going to be the 
re-reading of the file, as this operation will be completing in 
microseconds.

If you go back to 1.2.15 and strace Dovecot, do you see the same 
re-reading of the mbox files after each 20 or so messages?  This 
behavior may be normal.  I'm not familiar with the code or I'd have told 
you if this is the case.  Regardless, I doubt re-reading is the cause of 
the slowness.  There's got to be something else in the loop besides the 
file read eating up the CPU time.  Given it's 10 minutes for a 20MB file 
it would seem logical that a large wait is being inserted or Dovecot is 
waiting for a lock to expire--something along these lines.  I'd doubt 
Dovecot is actually doing anything most of this 10 minutes.

I think you need to find out what Dovecot 2.x is waiting on that 1.2.15 
wasn't.

-- 
Stan