The search time was no better with it on than off.
So I'm thinking I got something misconfigured somewhere. It seems IMAP may not be using solr to fetch results. But this would be odd since I definitely do see a big improvements in times with fts plugins turned on when using roundcube.
OK, I'm finally getting somewhere. Found this lead here: https://dovecot.org/pipermail/dovecot/2012-February/081514.html
"doveadm fts optimize" throws an error:
doveadm(root): Error: Couldn't drop privileges: User is missing UID (see mail_uid setting)
Checking /var/log/mail.err, I see a ton of these entries:
83353 Aug 25 14:53:22 email dovecot: imap(s)<26536>
I think this will be nailed once I figure out this issue.