We are using Sitecore 8.0 Update 2 and Solr 4.10.3.
Today we had a website crash because Solr went down and our listing pages and search related components weren't working. The issue turned out to be that Solr was using 99% of the RAM. The Solr server has 8 GBs of RAM and JVM has the same allocated to it. It has never been a problem and for the last 2 years. The average RAM/JVM usage was about 30-40% of the total. But it spiked in the last couple of days and only a server restart helped it come back up.
After restarting the server the JVM is growing by a 100 or so MB every few seconds and seems to peak at about 3-4 GB but then it goes down to below 1 GB right away. The CM is the only server that does any indexing and the traffic on there is rather below average since Friday. The CDs are also not showing any high traffic rather average traffic.
Has anyone come across this scenario before? Do we need to add more RAM and increase JVM on the Solr server?