How to rebuild the analytics index without rebuilding the entire reporting database?

Question

In Sitecore 8.1.3, is it possible to rebuild the analytics index without rebuilding the entire reporting database as specified in doc.sitecore.net?

I have an analytics index that is more or less broken, and as far as I know it is not possible to trigger a rebuild of the index.

From https://community.sitecore.net/technical_blogs/b/getting_to_know_sitecore/posts/rebuilding-the-sitecore-analytics-index i can see that there are workaround, but I am worried that it does not take care of everything?

Answer 1

Have recently implemented this module, works perfectly for me so far, give it a try. https://vohil.net/2017/05/16/rebuild-sitecore-analytics-index-without-re-building-reporting-database/

Answer 2

Analyzing the proposed approach

Let's look at the workaround code from the linked article:

ContentSearchManager.GetIndex("sitecore_analytics_index").Reset();
var poolPath = "aggregationProcessing/processingPools/live";
var pool = Factory.CreateObject(poolPath, true) as ProcessingPool;
var driver = MongoDbDriver.FromConnectionString("analytics");
var visitorData = driver.Interactions.FindAllAs<VisitData>();
var keys = visitorData.Select(data => new InteractionKey(data.ContactId, data.InteractionId));
foreach(var key in keys)
{
    var poolItem = new ProcessingPoolItem(key.ToByteArray());
    pool.Add(poolItem);
}

BTW, the correct way to get the analytics index is ContentSearchManager.GetAnalyticsIndex().

What this code does is:

1. It empties the Analytics index;
2. It loads all interactions from the Collection database;
3. It adds interaction IDs to the interaction processing pool.

I honestly think that even on a moderately sized instance, this is not the best idea for the following reasons:

• It will result in an attempt to re-aggregate all interaction data to the Reporting database. This will most likely result in:
  • Incorrect reporting numbers;
  • Aggregation exceptions—mainly due to attempts of inserting already records with duplicate IDs to the table Trail_Interactions.
• It will put approximately the same load on your Sitecore installation as a full Reporting DB rebuild. The same aggregation code will have to run, so why not execute it the right way?
• Adding interactions to the processing pool all at once will be a very heavy task—what if there are hundreds of thousands of visits? This may take quite a while when done from a single machine. On the other hand, xDB has a specialized infrastructure for making this process distributed as a part of the standard rebuild process.
• While this code is adding interactions to the processing pool, new interactions will occur on the site. You've just cleared the index, but the new website interactions will continue being indexed. Can you guarantee that you won't index something twice, or, on the opposite, miss some interactions? Again, xDB has a specialized "cutoff" mechanism in place to prevent such situations, but only if you stick to the standard rebuild process.

It looks like the approach will actually cover all data. Let's look at how data gets into the index:

Sitecore.ContentSearch.Lucene.Index.Analytics.config

<index id="sitecore_analytics_index" ...>
  ...
  <locations hint="list:AddCrawler">
    <crawler type="Sitecore.ContentSearch.Analytics.Crawlers.AnalyticsVisitCrawler, Sitecore.ContentSearch.Analytics">
      <CrawlerName>Lucene Visit Crawler</CrawlerName>
      <ObservableName>VisitAggregationObservable</ObservableName>
      ...
    </crawler>
    <crawler type="Sitecore.ContentSearch.Analytics.Crawlers.AnalyticsVisitPageCrawler, Sitecore.ContentSearch.Analytics">
      <CrawlerName>Lucene Visit Page Crawler</CrawlerName>
      <ObservableName>VisitPageAggregationObservable</ObservableName>
      ...
    </crawler>
    <crawler type="Sitecore.ContentSearch.Analytics.Crawlers.AnalyticsVisitPageEventCrawler, Sitecore.ContentSearch.Analytics">
      <CrawlerName>Lucene Visit Page Event Crawler</CrawlerName>
      <ObservableName>VisitPageEventAggregationObservable</ObservableName>
      ...
    </crawler>
    <crawler type="Sitecore.ContentSearch.Analytics.Crawlers.AnalyticsContactCrawler, Sitecore.ContentSearch.Analytics">
      <CrawlerName>Lucene Contact Crawler</CrawlerName>
      ...
      <observables hint="list:AddObservable">
        <observable>ContactAggregationObservable</observable>
        <observable>ContactChangeObservable</observable>
      </observables>
    </crawler>
    <crawler type="Sitecore.ContentSearch.Analytics.Crawlers.AnalyticsContactTagCrawler, Sitecore.ContentSearch.Analytics">
      <CrawlerName>Lucene Contact Tag Crawler</CrawlerName>
      ...
      <observables hint="list:AddObservable">
        <observable>ContactTagAggregationObservable</observable>
        <observable>ContactTagChangeObservable</observable>
      </observables>
    </crawler>
    <crawler type="Sitecore.ContentSearch.Analytics.Crawlers.AnalyticsAddressCrawler, Sitecore.ContentSearch.Analytics">
      <CrawlerName>Lucene Address Tag Crawler</CrawlerName>
      ...
      <observables hint="list:AddObservable">
        <observable>AddressAggregationObservable</observable>
        <observable>AddressChangeObservable</observable>
      </observables>
    </crawler>
  </locations>

As you can see, there are 9 observables defined. The way it works is, these crawlers subscribe to observables' events and add or modify index entries when the events are fired.

The observables are wired to fire as part of the <interactions> and <contacts> pipelines.

Sitecore.ContentSearch.Analytics.config

<pipelines>
  <interactions>
    <processor type="Sitecore.ContentSearch.Analytics.Aggregators.AnalyticsVisitAggregator, Sitecore.ContentSearch.Analytics">
      <param desc="name">VisitAggregationObservable</param>
    </processor>
    <processor type="Sitecore.ContentSearch.Analytics.Aggregators.AnalyticsVisitPageAggregator, Sitecore.ContentSearch.Analytics">
      <param desc="name">VisitPageAggregationObservable</param>
    </processor>
    <processor type="Sitecore.ContentSearch.Analytics.Aggregators.AnalyticsVisitPageEventAggregator, Sitecore.ContentSearch.Analytics">
      <param desc="name">VisitPageEventAggregationObservable</param>
    </processor>
    <processor type="Sitecore.ContentSearch.Analytics.Aggregators.AnalyticsContactAggregator, Sitecore.ContentSearch.Analytics">
      <param desc="name">ContactAggregationObservable</param>
    </processor>
    <processor type="Sitecore.ContentSearch.Analytics.Aggregators.AnalyticsContactTagAggregator, Sitecore.ContentSearch.Analytics">
      <param desc="name">ContactTagAggregationObservable</param>
    </processor>
    <processor type="Sitecore.ContentSearch.Analytics.Aggregators.AnalyticsAddressAggregator, Sitecore.ContentSearch.Analytics">
      <param desc="name">AddressAggregationObservable</param>
    </processor>
  </interactions>
  <contacts>
    <processor type="Sitecore.ContentSearch.Analytics.Aggregators.ContactChangeContactAggregator, Sitecore.ContentSearch.Analytics">
      <param desc="name">ContactChangeObservable</param>
    </processor>
    <processor type="Sitecore.ContentSearch.Analytics.Aggregators.ContactChangeContactTagAggregator, Sitecore.ContentSearch.Analytics">
      <param desc="name">ContactTagChangeObservable</param>
    </processor>
    <processor type="Sitecore.ContentSearch.Analytics.Aggregators.ContactChangeAddressAggregator, Sitecore.ContentSearch.Analytics">
      <param desc="name">AddressChangeObservable</param>
    </processor>
  </contacts>
</pipelines>

The processors in the <contacts> pipeline are only needed for registering changes in contacts. Initial contact additions are handled via the <interactions> pipeline.

This means that it is enough to manually add all interactions to a processing pool, as per the article. Separately adding all contacts to a pool is not required. All indexable data will be obtained solely through the <interactions> pipeline.

Recommendation

So all in all, the proposed approach with reaggregating all interactions will work for updating the index. But it will very likely lead to reporting data inconsistencies, server slowdowns, and log errors. It will not be better performing than the normal Reporting DB rebuild.

Hence, my recommendation is to stick to Sitecore's recommendations and rebuild your Reporting database when you want to rebuild the index.

Hopefully, in the future there will be a better way, but for now, there isn't one.

Answer 3

The second link you described is the way to rebuild the analytics index without rebuilding the entire reporting database.

But why should you not use the rebuild of the reporting database? It is always better to do the entire rebuild and reindex instead of just rebuilding your index. If you have somehow corrupt data in your reporting database, your index will be broken again. For that reason, I would recommend you to just go with the reporting database rebuild.