On 11/9/2011 10:16 AM, Alexander Chekalin wrote:
Thanks, Robert,
will take a look at.
What I'm afraid for is how database storage should be planned (storage, CPU, RAM, scaling when will be over-filled). When dealing with files (I'm using maildir)
Bingo. ^^^
Maildir is very likely a hug factor in your current slow search time. With a maildir search, every mail file must be opened and searched. How many total mail files are opened for each of your searches? Thousands? Tens of thousands? Maildir causes a massive disk IO bottleneck when searching so many files. Run iostat the next time you do one of these searches, and look at the %iowait value. It will likely be very high. If it is, this confirms maildir is a big part of the problem.
mbox, and mdbox, would be many many times faster than maildir WRT searching as the total number of files is lower by orders of magnitude. Switching from maildir to mbox/mdbox shifts the workload burden from the disk subsystem to the processor/memory. And I'm sure as with everyone else on the planet today, you have massive spare CPU cycles, but extremely limited spindle throughput.
And as Timo suggested, using one of the indexing search plugins would be much faster yet, as long as you keep the indexes updated.
-- Stan