I'm working with a customer right now that has seen their Solr implementation become unstable lately. Based on some initial analysis, signs are pointing towards the analytics index. They've been live with xDB on Sitecore 8 for almost a year now, so this index has grown throughout the year. Currently it contains 25 million documents, and continues to grow. We can throw more memory at the server but that's a short term fix until this index grows beyond the hardware's capacity again.
Right now the analytics index contains 25 million documents, consumes 51818368 of heap space, and is almost 11 GB on disk. The disk size will spike up to 30 GB as it merges, but we have plenty of disk space. RAM is another issue, that is being pushed to the limit. I'm also seeing in the resource monitor that this index is reading & writing at about 14 MB/s to different segments in the core. Nothing else comes anywhere near that.
We're investigating scalability options for the future, such as sharding with Solr Cloud, but the immediate problem is right now Solr will become unresponsive and needs to be restarted. When Solr fails, Sitecore fails. It's become an almost daily nuisance for the customer.
In 8.1 update 3 and 8.2, there is a setting to disable anonymous contact indexing, ContentSearch.Analytics.IndexAnonymousContacts. I think this will go a long way to reducing the pace at which this index grows.
In the short term, is there a way to safely reduce the size of this index? Can I run a query against the Solr core to remove these anonymous contacts? Really any advice offered on scaling and maintaining the analytics index would be appreciated.