Impacts of enabling the indexing of anonymous contacts in the xDB index?

Question

We're looking on enabling the indexing of the anonymous contacts in the xdb index, but the https://doc.sitecore.com/en/developers/91/sitecore-experience-platform/enable-indexing-of-anonymous-contacts-in-the-xdb-index.html page warns about the following:

Enabling this setting typically causes a significant increase in the amount of data to be indexed. Make sure that both Solr / Azure Cognitive Search and the xConnect Search Indexer are sized appropriately.

What is "sized appropriately"? Which setting is it referring to? Is it the ParallelizationDegree and BatchSize in the sc.Xdb.Collection.IndexerSettings.xml? What's a general approach to make sure that it's sized appropriately?

Answer 1

xConnect Search Indexer will either be a Windows Service or Azure WebJob. So, I believe the server in which this service or job is configured should be able to cope up with the processing load - which will be based on the number of visitors that come to your website.

As the index grows in Solr / Azure Cognitive Search, there is also a growth in the server disk space as described in this scenario.

Answer 2

First of all, when enabling indexing of anonymous contacts in the xDB index, it's important to keep in mind that the size of the index and the amount of data that needs to be processed will increase significantly. And it will increas not only index but also the size of the xDB SQL database. This is because for public websites, the majority of traffic comes from anonymous visitors, and enabling indexing for these visitors means that data will need to be collected and indexed for a much larger number of interactions.

In general, you can expect that the size of the xDB index and the amount of data that needs to be processed will approximately double when enabling indexing of anonymous contacts. This can have an impact on the performance and resource usage of both Solr / Azure Cognitive Search and the xConnect Search Indexer, and it's important to ensure that both are sized appropriately to handle the increased load.

Regarding the specific settings in the sc.Xdb.Collection.IndexerSettings.xml file, both ParallelizationDegree and BatchSize can affect the indexing performance and should be considered when sizing appropriately for the indexing load. The ParallelizationDegree setting determines how many parallel threads the xConnect Search Indexer uses to index data, while the BatchSize setting determines how many documents are processed in a single batch.

To ensure that Solr / Azure Cognitive Search and the xConnect Search Indexer are sized appropriately, you may want to consider the following general approach:

1. Review the documentation for Solr / Azure Cognitive Search and xConnect Search Indexer to understand the recommended hardware and software requirements.

2. Monitor the resource usage (such as CPU, memory, disk usage, and network bandwidth) on the servers hosting Solr / Azure Cognitive Search and xConnect Search Indexer to identify any bottlenecks or performance issues.

3. Test the indexing load with the new configuration settings and monitor the system resources to ensure that the system can handle the additional load.

4. Adjust the configuration settings as needed to optimize the performance and ensure that the system can handle the expected indexing load.

It's important to note that sizing appropriately for indexing can be a complex process that depends on many factors, including the size and complexity of your xDB data, the number of anonymous contacts you expect to index, and the overall system configuration.