Sitecore 10.1 xConnect collection database multi-region scaling read-only performance

Question

I have a multi-region Sitecore deployment with CDs in the US and the UK using Sitecore 8.2 with MongoDb as the xDb, like this:

UK                                US

CD1                               CD2
Mongo primary + arbiter  <----->  Mongo secondary

Mongo is geo-distributed with a connection string on each CD that prefers reads from the nearest replica but will always write to the primary replica. This ensures any kind of read of xDB contact information on page-load is as fast as possible for each region.

I want to upgrade to Sitecore 10.1. MongoDB has been discontinued for xDB (now xConnect) in 10.1 and I must use SQL server.

I have read up on read scale-out (https://doc.sitecore.com/developers/101/platform-administration-and-architecture/en/xconnect-scalable-reads.html) but this does not appear to be relevant to my use case - it's for same-region scale-out, not for geo-replication.

The official docs (https://doc.sitecore.com/developers/101/platform-administration-and-architecture/en/scaling-and-configuring-the-xconnect-collection-service-role.html) state that:

Deploy the xConnect Collection service as close as possible to the xDB Collection database. It is more important to reduce latency between the service and the database than between the service and clients such as Content Delivery.

They also state:

All instances must read from a single centrally located xDB Collection database (database sharding by geography is not supported).

The docs don't state what kind of performance hit you're going to take by the CD communicating to the xConnect service sat in a separate region.

Am I right in thinking that there will be a performance hit due to the db not being geo-replicated? Or is there some other mechanism I'm missing that allows Sitecore to load a contact profile for personalization on page-load? If not, how can I geographically replicate the xConnect collection SQL database in order to minimize read latency on page load within each region?

EDIT I'm specifically talking about the following sequence of events (particularly points 2 and 4) as described in the docs here (https://doc.sitecore.com/developers/101/sitecore-experience-platform/en/tracking-contacts.html)

1. Contact visits www.sitecore.net - they have the SC_ANALYTICS_GLOBAL_COOKIE set to 60573a38-6925-4d92-8bff-327e41650d8f

2. Request is sent to xConnect for the device profile matching 60573a38-6925-4d92-8bff-327e41650d8f

3. The LastKnownContactId of the device profile is used to retrieve a contact from xConnect by its tracker identifier

4. If the contact exists, it is loaded into session - if not, a new GUID is generated (the new contact will be saved on session end)

5. At the end of the session, a second interaction is saved against the contact

Answer 1

The performance hit isn't in the documentation because it depends on a myriad of factors. The key thing to understand here is latency. There are various ways of mitigating this latency however. You can have xConnect Collection role services in each region and use xConnect Scalable Reads combining them with Azure SQL Auto Failover Group capabilities.

I will warn you however that the biggest impact does not come from your xConnect Collection, but from xConnect Reference Data. Even if you choose to have only 1 region with the xConnect databases and Collection service I highly recommend you replicate the xConnect Reference Data service to each regional data center. This service makes heavy use of caching so it will speed up access without needing DB replication and additionally when this service is unavailable your CD will also fail to serve requests.

How does this work?

When you activate Scalable Reads under the hood the SQLProvider of xConnect will start adding ApplicationIntent to the connectionstring it receives from the SMM database:

So what does this mean when combining with Azure Auto Failover Groups? Consider the following:

Your failover group has both a Read-write listener and a Read-only listener that will automatically direct your requests along the most optimal and desired path. Thanks to the included ApplicationIntent added by scalable reads you don't need to do anything, Azure and Sitecore are doing everything under the hood for you.

How to set this up?

Make sure that inside the SMM database your connection strings are targeting your failover groups.

Answer 2

I solved this by using a reverse proxy in the following architecture (apologies for poor Visio skills):

The key things to note here are as follows...

Database setup

• The xConnect database is actually 3+ databases: a ShardMapManager plus 2 (or more) shard databases. The default is 2.
• The ShardMapManager db contains the table [__ShardManagement].[ShardsGlobal] where the actual server name of the shards is stored. You can edit the Server column manually to point to the right server (Sitecore does this for you when you set up xConnect using the Sitecore.Xdb.Collection.Database.SqlShardingDeploymentTool.exe but it's easy to change).
• I deployed the ShardMapManager db in my main region (UK) and then copied it to my secondary region (US) manually. Note, it is NOT geo-replicated.
• On the secondary US ShardMapManager db I updated the [__ShardManagement].[ShardsGlobal].[Server] column to reflect the US server name.
• I then simply geo-replicated the 2 shard dbs into the same US server.

xConnect setup

• I deployed 2 instances of xConnect, 1 in each region pointing to their respective ShardMapManager dbs.
• The US xConnect instance cannot function fully - it will throw errors if it tries to perform any write operation to the shard dbs, since the shards are geo-replicated read-only replicas.
• The UK xConnect instance is fully functional with read/write access to the shard dbs.
• I pointed my UK CDs directly at the UK r/w xConnect instance.
• I deployed a reverse proxy in the US region which routes to either the UK xConnect instance if the HTTP method = POST, or the US xConnect instance if the HTTP method = GET. This is the most crucial piece of the puzzle, and prevents any read-only-based errors occurring on the US xConnect instance
• The reverse proxy is an nginx web server running on Linux on a VM. I tried to achieve the same with a PaaS solution using Azure Application Gateway but you cannot route based on HTTP Method which was frustrating. The nginx config I'm using is below.
• I pointed my US CDs at the nginx reverse proxy.

TLS/SSL Considerations

• xConnect requires client certificates for communication between itself and the CDs.
• The introduction of nginx complicates this, because nginx must use the same client certificate as the CDs to communicate with both xConnect instances.
• I attempted to get nginx to forward the client certificate from the CD to xConnect but I couldn't get it to work.
• I solved the problem by installing the client certificate on the nginx VM, and using it in my config as you'll see below.
• nginx must also have its own server certificate just like xConnect does. Since my nginx VM has no public IP or fqdn, and I didn't want to faff around with private DNS zones, I generated a self-signed certificate where the CN was just the hostname of the VM. Of course this is not trusted by the US CD, so Sitecore throws SSL errors when communicating with xConnect. To get around this, I installed the self-signed nginx cert into the "Trusted Root Certificate Authorities" on each US CD VM.

Here is the nginx config:

    server {
        listen 443 ssl;
        listen [::]:443 ssl;
        ssl_certificate /etc/ssl/private/{self-signed server cert}.crt;
        ssl_certificate_key /etc/ssl/private/{self-signed server cert}.key;
        ssl_protocols  TLSv1.2;

        server_name {vm hostname};

        location / {
                if ($request_method != 'GET'){
                        proxy_pass https://{UK read-write xConnect}.azurewebsites.net;
                }

                        proxy_pass https://{US read-only xConnect}.azurewebsites.net;
                        proxy_http_version 1.1;
                        proxy_set_header Upgrade $http_upgrade;
                        proxy_set_header Connection keep-alive;
                        proxy_set_header X-Forwarded-Proto https;
                        proxy_set_header X-Forwarded-Port $server_port;
                        proxy_set_header X-Forwarded-For $remote_addr;
                        proxy_ssl_certificate /etc/ssl/private/{xConnect client cert}.crt;
                        proxy_ssl_certificate_key /etc/ssl/private/{xConnect client cert}.key;
                        proxy_connect_timeout       300;
                        proxy_send_timeout          300;
                        proxy_read_timeout          300;
                        send_timeout                300;

        }
    }

    server {
            listen 80;
            listen [::]:80;

            server_name  {vm hostname};
            return 302 https://$server_name$request_uri;
    }

Answer 3

From understanding above you might need to think in Azure Apps way rather than Sitecore.

As all the xDB/xConnect logical role are web apps incase of performance and distributed multi region/ Geo distributed application/environments can be setup by

App-service-environment-geo-distributed-scale

And for deployment you might can use this