[Dovecot] Please advise on very fast search

[Stan Hoeppner]

On 11/9/2011 10:16 AM, Alexander Chekalin wrote:
> Thanks, Robert,
> 
> will take a look at.
> 
> What I'm afraid for is how database storage should be planned (storage,
> CPU, RAM, scaling when will be over-filled). When dealing with files
> (I'm using maildir)

Bingo.        ^^^

Maildir is very likely a hug factor in your current slow search time.
With a maildir search, every mail file must be opened and searched.  How
many total mail files are opened for each of your searches?  Thousands?
 Tens of thousands?  Maildir causes a massive disk IO bottleneck when
searching so many files.  Run iostat the next time you do one of these
searches, and look at the %iowait value.  It will likely be very high.
If it is, this confirms maildir is a big part of the problem.

mbox, and mdbox, would be many many times faster than maildir WRT
searching as the total number of files is lower by orders of magnitude.
 Switching from maildir to mbox/mdbox shifts the workload burden from
the disk subsystem to the processor/memory.  And I'm sure as with
everyone else on the planet today, you have massive spare CPU cycles,
but extremely limited spindle throughput.

And as Timo suggested, using one of the indexing search plugins would be
much faster yet, as long as you keep the indexes updated.

-- 
Stan