Please check the current utilization of JVM Memory of your Solr Instance.
I faced this once in the past, when my JVM Memory in Solr Dashboard was showing to have consumed 95% and everything was getting slow, even the results on the page. If this is the case, increase the Solr JVM Memory to consume more RAM and this might be faster.
Technically, what happens is because the Memory is already occupied by other indexes, rebuild might get a very small chunk of the RAM and hence be slow.
Check if the same helps. I faced this challenge with a project in the past, with Solr Cloud in the architecture. Here is an article, might not relate to you directly, but might help in understanding how it worked https://blog.varunvns.in/2019/09/10/solr-cloud-is-super-slow-or-not-responding-search-functionalities-not-working/
If you need to increase the JVM Memory, you can check out the following Stack Exchange link: How to increase Solr JVM memory
Hope this helps.