How can I reduce the size of the analytics index? [alt: Populating the Analytics Index]

Question

I'm working with a customer right now that has seen their Solr implementation become unstable lately. Based on some initial analysis, signs are pointing towards the analytics index. They've been live with xDB on Sitecore 8 for almost a year now, so this index has grown throughout the year. Currently it contains 25 million documents, and continues to grow. We can throw more memory at the server but that's a short term fix until this index grows beyond the hardware's capacity again.

Right now the analytics index contains 25 million documents, consumes 51818368 of heap space, and is almost 11 GB on disk. The disk size will spike up to 30 GB as it merges, but we have plenty of disk space. RAM is another issue, that is being pushed to the limit. I'm also seeing in the resource monitor that this index is reading & writing at about 14 MB/s to different segments in the core. Nothing else comes anywhere near that.

We're investigating scalability options for the future, such as sharding with Solr Cloud, but the immediate problem is right now Solr will become unresponsive and needs to be restarted. When Solr fails, Sitecore fails. It's become an almost daily nuisance for the customer.

In 8.1 update 3 and 8.2, there is a setting to disable anonymous contact indexing, ContentSearch.Analytics.IndexAnonymousContacts. I think this will go a long way to reducing the pace at which this index grows.

In the short term, is there a way to safely reduce the size of this index? Can I run a query against the Solr core to remove these anonymous contacts? Really any advice offered on scaling and maintaining the analytics index would be appreciated.

Answer 1

In general Solr usage, there are a few things you could do in order to tweak how data is stored in the index. In terms of Sitecore, I am not fully sure if the exact same rules apply, due to the way Sitecore internally might work with the indexes - so you may want to be a bit careful, when trying some of these things out and perform a thorough test for each tweak you apply.

In your Solr core for the analytics core, find and open the schema.xml file. In this file, you could try to:

1. Locate all fields that are being stored and try remove those stored fields from the index. Once done, try to let Sitecore make a query to the Solr index for the data it needs using the contacts API, and verify that it gets correct data back.

2. Add omitNorms="true" to text fields that don't need length normalization. From the Solr wiki (http://wiki.apache.org/solr/SchemaXml):

   Set to true to omit the norms associated with this field (this disables length normalization and index-time boosting for the field, and saves some memory). Only full-text fields or fields that need an index-time boost need norms.

3. Add omitPositions="true" to text fields that don't require phrase matching. This might be a bit tricky to test out, since you'd need to know how Sitecore internally works with the given fields.

You can see the performance benchmarks found and described in this article http://css.dzone.com/news/solr-index-size-analysis, where omitting the norms and positions shows it will help you save a lot of space in terms of the index size.

As said in the beginning of my answer, this may or may not work in terms of the way Sitecore works with the index, but it's worth a try to see if a bit of tweaking can solve your problems, as this is how you could solve the issue when working in a non-Sitecore context.

Answer 2

There's no ready-made way to reduce the index - short of rebuilding it without the documents you are trying to exclude. You're definitely facing a custom task here. Contacts are indexed by an "observer". Unlike a crawler, this attaches itself to events on the contact object and updates trigger writes to the index.

We have built a hack process that loops the contacts and then "touches" (double-entendre intentional) each one in an attempt to rebuild this index when we noticed there were some holes. The code is shown below - obviously you need to modify the loop to have it skip those contacts you don't want indexed. If I were you I would set this up on a non-prod machine somewhere are connect it to the production MongoDBs and a completely new SOLR core that you can build without disrupting the production environment. It will take a looong time to run if you have millions of contacts.

<%@ Page Language="C#" Buffer="false" %>

<%@ Import Namespace="System" %>
<%@ Import Namespace="System.Linq" %>
<%@ Import Namespace="System.Threading" %>
<%@ Import Namespace="MongoDB.Bson" %>
<%@ Import Namespace="MongoDB.Driver" %>
<%@ Import Namespace="MongoDB.Driver.Builders" %>
<%@ Import Namespace="Sitecore.Analytics.Aggregation.Pipelines.ContactProcessing" %>
<%@ Import Namespace="Sitecore.Analytics.Data" %>
<%@ Import Namespace="Sitecore.Data" %>
<%@ Import Namespace="Sitecore.Pipelines" %>

<script runat="server">
    public class Contact
    {
        public BsonBinaryData _id { set; get; }
    }

    public void CallMain()
    {
        var client = new MongoClient("mongodb://mongo1.australiaeast.cloudapp.azure.com,mongo2.australiaeast.cloudapp.azure.com,mongo3.australiaeast.cloudapp.azure.com/admin?readPreference=primary");
        var db = client.GetServer().GetDatabase("analytics");
        var coll = db.GetCollection<Contact>("Contacts");

        var filter = Query.Exists("Identifiers.Identifier");
        var count = 0;

        var cursor = coll.Find(filter);
        cursor.SetBatchSize(2000);
        cursor.SetFields(Fields.Include("_id"));

        Response.Write( "<h2>Total count is: " + cursor.Count() + "</h2>");
        Response.Write( "<h3>Processing started at " + DateTime.Now.ToString() + "</h3>" );

        foreach(var doc in cursor)
        {             
            count++;
            Response.Write( "<p>" + count + "; " + DateTime.Now.ToString("HH:mm:ss") + "<span style='color:silver'>: " + doc._id.AsGuid + "; ");
            var pipelineArgs = new ContactProcessingArgs(new ID(doc._id.AsGuid), ProcessingReason.Updated);

            try  {
                CorePipeline.Run("contacts", pipelineArgs, "analytics.aggregation", true);//Pipeline.Start("analytics.aggregation/contacts", pipelineArgs);
            } catch( Exception e ) {
                Response.Write("<p><span style=\"background-color:lemonchiffon;color:darkred;border:1px solid darkred\">ERROR: " + e.Message + " - ");
                return;
            }
            Response.Write("contact: " + pipelineArgs.GetContact().Identifiers.Identifier + " - </span></p>" );

            if ( count % cursor.BatchSize == 0 )
            {
                Response.Flush();
                Thread.Sleep(10);
            }
        }
        Response.Write( "<h3>Processing finished at " + DateTime.Now.ToString() + "</h3>" );
    }
</script>
Touching Contacts:
<% CallMain(); %>

Put the above into an ASPX file in the root of the site, then just load the page and watch the magic happen.