Dovecot and FTS experiment
Hello,
I'm trying to experiment with Dovecot and Solr server. I have >30k email addresses that I want to index to speed up searching and save IOPS on mail servers. For now - I'm doing some experiments and I'm testing how it is working. I'm thinking about adding one additional server with Solr and configure all mail servers to use that server.
I have some questions. dir has 5MB. After indexing mailbox in Solr - index files has 15MB. What
- I have 15 mail servers. It will be good If I add new server with Solr and use it on all Dovecot servers? Or maybe I should install Solr on all mail servers?
- I notice - I have mail account with 3GB of mail. Index files in mail
changes in those files? FTS indexing adds something to that files - but what? 3. If I need to disconnect Dovecot from Solr server - simply Can I remove plugin configuration from dovecot.conf and everything would be working? Or maybe - I need to remove index files and recreate it? It is important for me because if some day I would decide to disconnect FTS server - I would have to recreate indexes for >30k mailboxes.. 4. What if some day I want to change Solr server to new server with empty Solr database? I can simply reindex mailboxes? Or I need to do something more?
Thanks!
On 29 January 2019 at 19:15 Tomasz Nowak tomek@flup.pl wrote:
Hello,
I'm trying to experiment with Dovecot and Solr server. I have >30k email addresses that I want to index to speed up searching and save IOPS on mail servers. For now - I'm doing some experiments and I'm testing how it is working. I'm thinking about adding one additional server with Solr and configure all mail servers to use that server.
I have some questions.
- I have 15 mail servers. It will be good If I add new server with Solr and use it on all Dovecot servers? Or maybe I should install Solr on all mail servers?
Possibly, it's hard to say. It depends more about how much mail you have in total. You might need more than one server. One per backend is definetly too much.
- I notice - I have mail account with 3GB of mail. Index files in mail dir has 5MB. After indexing mailbox in Solr - index files has 15MB. What changes in those files? FTS indexing adds something to that files - but what?
You can use doveadm dump /path/to/index/directory to view the contents on the indexes.
- If I need to disconnect Dovecot from Solr server - simply Can I remove plugin configuration from dovecot.conf and everything would be working? Or maybe - I need to remove index files and recreate it? It is important for me because if some day I would decide to disconnect FTS server - I would have to recreate indexes for >30k mailboxes..
You can just stop loading fts.
- What if some day I want to change Solr server to new server with empty Solr database? I can simply reindex mailboxes? Or I need to do something more?
You can run doveadm fts rescan -A and doveadm index -A "*" to perform full FTS indexing.
Thanks!
Aki
W dniu 2019-01-29 19:37, Aki Tuomi napisał(a):
Possibly, it's hard to say. It depends more about how much mail you have in total. You might need more than one server. One per backend is definetly too much.
Great, thank you. I have more than 15TB of mails and I know that one Solr server could be not enough. But it is good that I not need to install Solr od every mail server. On that amount of data - I think that Solr server need more optimalization and I don't want to change configuration of mail servers.
You can use doveadm dump /path/to/index/directory to view the contents on the indexes.
Great command, thank you! I didn't know about it.
You can just stop loading fts.
Super. I like it :)
You can run doveadm fts rescan -A and doveadm index -A "*" to perform full FTS indexing.
Ok, thank you. I was afraid that there might be more dependencies.
On 1/29/2019 9:15 AM, Tomasz Nowak wrote:
Hello,
I'm trying to experiment with Dovecot and Solr server. I have >30k email addresses that I want to index to speed up searching and save IOPS on mail servers. For now - I'm doing some experiments and I'm testing how it is working. I'm thinking about adding one additional server with Solr and configure all mail servers to use that server.
I have some questions.
- I have 15 mail servers. It will be good If I add new server with Solr and use it on all Dovecot servers? Or maybe I should install Solr on all mail servers?
You need to start somewhere. If you've never played with Solr before I suggest you start with one and get it working before you explore "sharding". When you're ready for that you should consult the solr mailing list. The importance of enough RAM for Solr cannot be overstated.
- I notice - I have mail account with 3GB of mail. Index files in mail dir has 5MB. After indexing mailbox in Solr - index files has 15MB. What changes in those files? FTS indexing adds something to that files - but what?
What mail storage format are you using? dbox?
Thinking...I believe that Dovecot records which mails have been reported to the FTS. That may help account for the increased size.
-- Daniel
W dniu 2019-01-29 23:48, Daniel Miller via dovecot napisał(a):
You need to start somewhere. If you've never played with Solr before I suggest you start with one and get it working before you explore "sharding". When you're ready for that you should consult the solr mailing list. The importance of enough RAM for Solr cannot be overstated.
Ok. Thank you. It's good to know that everything will work good with external Solr server. Most of the howto's that I read - describes local installation of every component, so I need to ensure that there wont be problems.
- I notice - I have mail account with 3GB of mail. Index files in mail dir has 5MB. After indexing mailbox in Solr - index files has 15MB. What changes in those files? FTS indexing adds something to that files - but what?
What mail storage format are you using? dbox?
Thinking...I believe that Dovecot records which mails have been reported to the FTS. That may help account for the increased size.
Today - maildir. But I have dbox migration in TODO. After another answear I know "doveadm dump" command, so I can check what are in indexes now.
participants (3)
-
Aki Tuomi
-
Daniel Miller
-
Tomasz Nowak