[Dovecot] Squat indexing a Maildir of over 600 GB?

[John Wells]

On Mon, Jul 21, 2008 at 1:20 PM, Timo Sirainen <tss at iki.fi> wrote:
> On Mon, 2008-07-21 at 12:55 -0400, John Wells wrote:
>> On Mon, Jul 21, 2008 at 12:50 PM, Timo Sirainen <tss at iki.fi> wrote:
>> > On Mon, 2008-07-21 at 12:37 -0400, John Wells wrote:
>> >> Guys,
>> >>
>> >> We have a very large maildir for email auditing purposes. It's
>> >> currently at 600 GB and continues to grow.
>> >>
>> >> Can dovecot handle this with squat indexing, or am I out of my mind?
>> >
>> > You can try of course, but that might be a bit too much. :) I've only
>> > tested with a 1,4 GB mailbox and memory usage went somewhere like 700 MB
>> > I think.
>>
>> Aha...I see...I was under the mistake impression that this was a
>> disk-based index.
>
> It's stored on disk, but when indexing it needs to keep parts of the
> index in memory.
>
>> Given that squat seem unfeasible, can anyone recommend another
>> approach? I'll look at Lucene integration, but if anyone knows of a
>> dovecot way or of another tool that would do this effectively,
>> commercial or open source, please let me know.
>
> v1.1.2 has Solr support. It might work:
> http://wiki.dovecot.org/Plugins/FTS/Solr

Thanks Timo...from what I know of Solr, it can handle it. But I'm
curious how the integration works...specifically:

- When are messages added to Solr? Is it only when new ones arrive, or
can older messages be injected as well?
- How does searching work? Do you need a front-end search tool to Solr?

Thanks!
John