[Dovecot] Please advise on very fast search

[Stan Hoeppner]

On 11/9/2011 11:35 PM, Alexander Chekalin wrote:
> Hello, Stan,
> 
> in fact the only thing I miss even with my current scheme is permanent
> ID assigned to the message so I can easily find it despite the IMAP
> mailbox it is now (so if someone moved the message from one
> mailbox/folder to another, the ID allows to retrieve it fast anyway).
> 
> You see, what I need is not only find message from|to someone on
> specified date, I also sometime need to restore that message back to
> user's original box. As far our mailserver and backup-mailserver are
> different machines, it is a bit tricky to copy messages between it fast
> enough. Say, if I need to find and restore all mails from
> user at domain.com within 2009 year, and search yields in some 1000's of
> messages, then use IMAP to copy it over to another server takes some
> time - and if you consider both search time and restore/copy time the
> whole process may take "ages".

Apparently I didn't fully understand all of your requirements.

Moving the archived mail to mbox/mdbox and/or getting a good indexing
search engine installed will cut the search time down tremendously.
Whether that would make up for the time consumed with an IMAP copy of
many emails I don't know.  If your servers aren't old and slow, and are
not already overloaded, I would think the IMAP message copying over GbE
would be pretty quick, even for the 1000 messages scenario.

There may be some Dovecot tweaks that might make this copy process
faster.  Timo would need to chime in on that.  Do you perform the IMAP
transfers with a GUI IMAP client on your management PC?  Or are you
using imapsync or some other util directly on the servers?

If the former you may be able to tweak your IMAP client to speed up the
transfers as well.  Try using IMAP and not IMAPS for the transfers.
What is the network infrastructure between the servers and your
management workstation?  Is it all GbE with jumbo frames enabled?

> With maildir I can rsync/scp needed files to another host and that's
> fast way - that's why I stick with maildir.

There is definitely some flexibility here.

> FTS in my case can help (I can search for user at domain.com, for example),
> but it also return messages that contains such a string in message body
> (and that takes index space, too), so I'll need to filter it later, but
> surely it'll be faster than checking every message in the archive.

Sure.  So you're concerned with your poor performance, but also with
disk space.  Unfortunately there's no free lunch to be had.  You'll have
to make sacrifices somewhere.  You could go with mdbox and use
compression, trading that saved space for search index files space.

-- 
Stan