[Dovecot] Please advise on very fast search

[Alexander Chekalin]

Oh, that's the point to consider. 

But I must confess I'm in love with Maildir for maybe 10 years for that simple fact I can do anything with each and every single message even on disk (=much faster than via IMAP). If I would deal with mbox directly I'd need to parse huge files, brrrr.

Are there any ways I can search or parse mboxes or mdboxes not directly and not with IMAP (I'm afraid it slooow in dump parsing)?

10.11.2011, в 3:42, Stan Hoeppner <stan at hardwarefreak.com> написал(а):

> On 11/9/2011 10:16 AM, Alexander Chekalin wrote:
>> Thanks, Robert,
>> 
>> will take a look at.
>> 
>> What I'm afraid for is how database storage should be planned (storage,
>> CPU, RAM, scaling when will be over-filled). When dealing with files
>> (I'm using maildir)
> 
> Bingo.        ^^^
> 
> Maildir is very likely a hug factor in your current slow search time.
> With a maildir search, every mail file must be opened and searched.  How
> many total mail files are opened for each of your searches?  Thousands?
> Tens of thousands?  Maildir causes a massive disk IO bottleneck when
> searching so many files.  Run iostat the next time you do one of these
> searches, and look at the %iowait value.  It will likely be very high.
> If it is, this confirms maildir is a big part of the problem.
> 
> mbox, and mdbox, would be many many times faster than maildir WRT
> searching as the total number of files is lower by orders of magnitude.
> Switching from maildir to mbox/mdbox shifts the workload burden from
> the disk subsystem to the processor/memory.  And I'm sure as with
> everyone else on the planet today, you have massive spare CPU cycles,
> but extremely limited spindle throughput.
> 
> And as Timo suggested, using one of the indexing search plugins would be
> much faster yet, as long as you keep the indexes updated.
> 
> -- 
> Stan