Random guess... Buffering? Whatever is sending to the browser isn't sending enough bytes to flush the buffer so the data is left in limbo until enough time goes by the buffer gets flushed anyways. Maybe a apache/nginx thing, php thing or browser thing. Remember its solr > dovecot > php > web server > browser. Tried other browsers?
I'm inclined to believe the problem is not that high up the food chain. Because when I query IMAP on a single folder over telnet following the instructions found here: https://doc.dovecot.org/configuration_manual/fts/solr/, imap reports that it's taking 3 to 4 seconds to return results:
a search text "maynez" a OK Search completed (3.386 + 0.001 + 0.250 secs).
This particular search returns a few hundred results.
The speed of the query changed depending on whether a lot of results are returned. So if I search on a nonsensical word like "zyzzix", imap reports results nearly instantaneously:
a search text "zyzzix" a OK Search completed (0.012 + 0.000 + 0.006 secs).