I'm inclined to believe the problem is not that high up the food chain. Because when I query IMAP on a single folder over telnet following the instructions found here: https://doc.dovecot.org/configuration_manual/fts/solr/, imap reports that it's taking 3 to 4 seconds to return results:
a search text "maynez" a OK Search completed (3.386 + 0.001 + 0.250 secs).
This particular search returns a few hundred results.
The speed of the query changed depending on whether a lot of results are returned. So if I search on a nonsensical word like "zyzzix", imap reports results nearly instantaneously:
a search text "zyzzix" a OK Search completed (0.012 + 0.000 + 0.006 secs).
I just did a quick test. I did a search over imap over telnet with fts plugins turned off and with them turned on.
The search time was no better with it on than off.
So I'm thinking I got something misconfigured somewhere. It seems IMAP may not be using solr to fetch results. But this would be odd since I definitely do see a big improvements in times with fts plugins turned on when using roundcube.