XConnect IndexWorker error "Tokens are incompatible, they have different set of shards."

Question

The log for my IndexWorker is filling up with the following error:

Tokens are incompatible, they have different set of shards

Any ideas what might be causing this?

Answer 1

Credit to Rob Earlam for helping me find the solution to this.

Essentially, information about the shard databases is stored in the xdb index. If either the server or database name of your shards changes, this will result in this error being logged. In my case, we had cloned (what was now) "production" to be the new "UAT" but the index still referred to the production database server.

To confirm if this is indeed your problem, you can do the following (skip ahead if you're already sure):

In Solr, query the Sitecore_xdb index for id:xdb-index-token. Next, copy the xdbtokenbytes_s value and provide it to the following PowerShell script:

ConvertFrom-XdbIndexToken.ps1

param(
    [string]$Token = (Read-Host "Enter xdbtokenbytes_s value")
)

$bytes = [Convert]::FromBase64String($Token)
$ms = [System.IO.MemoryStream]::new($bytes, 0, $bytes.Length)
$bf = [System.Runtime.Serialization.Formatters.Binary.BinaryFormatter]::new()
$dict = $bf.Deserialize($ms);

Write-Output $dict

It will output something like this:

[DataSource=DBSERVERNAME Database=Sitecore_Xdb.Collection.Shard0] - 1739
[DataSource=DBSERVERNAME Database=Sitecore_Xdb.Collection.Shard1] - 1349

If the indexed values don't match the actual db Server/Database, you'll need to rebuild the XDB index. There is documentation available on how to do this, but it will usually boil down to running the following on your xconnect server:

XConnectSearchIndexer.exe -requestrebuild

Original C# version

If PowerShell isn't an option, here's the equivalent in C#. Run it in a console app (or via a service like .NET Fiddle)

using System;
using System.Collections.Generic;
using System.IO;

public class Program
{
    public static void Main()
    {
        var s = "TOKEN_HERE";
        byte[] bytes = Convert.FromBase64String(s);
        using (var rs = new MemoryStream(bytes, 0, bytes.Length))
        {
            var sf = new System.Runtime.Serialization.Formatters.Binary.BinaryFormatter();
            var o = sf.Deserialize(rs) as Dictionary<string, long?>;
            foreach(var kvp in o)
            {
                Console.WriteLine($"{kvp.Key} - {kvp.Value}");
            }
        }
    }
}

Answer 2

In case this error persists, even after you rebuild the XDB index, you might as well want to check the items under /sitecore/system/List Manager/ListOperations node.

These items are created by the List Manager application and they contain a field named Sync Token. You should update the Sync Token field of each item with the one corresponding to your local instance. These items will eventually disappear as the List Manager process them.

To find out what Sync Token you should use, just do as Richard Szalay suggested above. In Solr, query the [prefix]_xdb index for id:xdb-index-token field.