xConnect in a DR scenario

Question

NOTE: In our environment, SQL is cloud hosted. It has its own fail over strategy which is separate from the server/site DR.

NOTE: Our DR and Production environments are in different geo zones. We can't share a SOLR instance as they are running in the same zones as the sites. We have not looked at any hosted SOLR services.

I am looking to upgrade a client from 8.2 to 9.1.1 that has a cold DR environment. We only bring this environment up to deploy to it with CI. DR with CM/CD is pretty simple, no issues there. Our DR task is to bring DR up, rebuild the indexes and switch the DNS.

But for xConnect it is more complicated.

xDB SOLR indexes

The first issue is that xConnect relies on the xDB index to keep track of all the profiles in Experience Profile and Experience Analytics. To get this up to date on DR, you would need to rebuild the xDB index with XConnectSearchIndexer -rr. The shards are in the shared SQL so the actual tracking data is fine.

Index Worker Service

Second issue is the index worker service. In the image below you can see the index worker service is not scalable. At least I don't think it is. But this would mean to be that we would not want prod and DR indexer services running at the same time. My guess is that the worker pool records do not have a column to know what record is currently in process. This would cause two processes to work on the same interaction and turn the analytics data to garbage.

So my guess here is that prod xConnect needs to be completely stopped on prod before it is started on DR. And I would need to disable the Sitecore windows services so they do not start on boot. Adds more complexity to fail over and testing.

https://doc.sitecore.com/developers/90/platform-administration-and-architecture/en/scale-vertically.html

Does anyone have any real world experience with xConnect in a DR scenario?

Answer 1

The best guidance from Sitecore that is available for Disaster Recovery scenarios at time of writing is Sitecore Managed Cloud Standard — Disaster Recovery.

Notes of interest from the article:

xConnect Search Indexer

Sitecore can only have one active xConnect Search Indexer WebJob across a solution. In case of any failover and restore of service, the indexer must be shut down.

xDB is excluded from RTO (Recovery Time Objective)

For Disaster Recovery, the RTO does not cover the xDB rebuild due to the significant amount of time it can take for a large content database. If the analytics indexes are not rebuilt this should only affect functionality that depends on lists (for example, EXM) and should not affect the frontend site.

Of course, in your question you have specified that you are not on Azure PaaS (you are IaaS), so some of it will not apply, but from this I gather that Sitecore's process for a failover is as follows:

1. If it's a cold failover they will clone the current services from the primary region to the secondary region. If a warm failover, these will already be cloned - but shut down.
2. They'll shut down the xConnect Search Indexer on the primary region and start it up along with the rest of the services in the secondary region.
3. Presumably once the xDB shards have been failed over successfully they will then rebuild the indexes in the secondary region. They note that this is not a priority as most sites will have a large about of data in xDB so it will likely take some time.

xConnect Search Indexer cannot be scaled, and when run as an Azure WebJob it takes advantage of the Singleton locking functionality.

So to summarise what you need to consider for xConnect in a DR scenario:

• You need to ensure that only one instance of the xConnect Search Indexer is ever running against a xDB index.
• For the most part all the xConnect roles (collection, refdata, search, cortex processing and reporting, marketing automation ops and reporting) are fairly set and forget. If you have any custom collection models (facets) or other config then you'll need to ensure those are kept in sync across the regions.
• RefData gets populated primarily via the Deploy Marketing Definitions tool so you'll need to kick that off when doing a fail-over unless you already have the RefData DB replicated.
• You'll need to have some kind of redundancy on your search service (e.g. Solr Cloud in your case) so that it doesn't become a single point of failure.
• Definitely do a DR failover test so you know how long it takes to get things up and running and know if there's any holes in the implementation. You could have some scripts to help with the process such as kicking off the Marketing Defs deployment and xDB reindex.