XConnect: An error occured during resuming a previous index rebuild process

Question

Does anyone have an idea about what could be causing this error? It happens after doing a re-build of the XConnect index. This is in Sitecore 9.3

2020-03-30 20:11:02.896 -05:00 [Error] An error occured during resuming a previous index rebuild process. There will be another attempt to check. The check interval after an error is 00:00:20.
Newtonsoft.Json.JsonSerializationException: Error converting value "{xdbtokenbytes_s=, id=0, xdbiscc_b=true}" to type 'Sitecore.Xdb.Collection.Search.Solr.SolrDocuments.CursorSnapshotDocument'. Path '[0]'. ---> System.ArgumentException: Could not cast or convert from System.String to Sitecore.Xdb.Collection.Search.Solr.SolrDocuments.CursorSnapshotDocument.
   at Newtonsoft.Json.Utilities.ConvertUtils.EnsureTypeAssignable(Object value, Type initialType, Type targetType)
   at Newtonsoft.Json.Utilities.ConvertUtils.ConvertOrCast(Object initialValue, CultureInfo culture, Type targetType)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.EnsureType(JsonReader reader, Object value, CultureInfo culture, JsonContract contract, Type targetType)
   --- End of inner exception stack trace ---
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.EnsureType(JsonReader reader, Object value, CultureInfo culture, JsonContract contract, Type targetType)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateList(IList list, JsonReader reader, JsonArrayContract contract, JsonProperty containerProperty, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateList(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, Object existingValue, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
   at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
   at Newtonsoft.Json.Linq.JToken.ToObject(Type objectType, JsonSerializer jsonSerializer)
   at Newtonsoft.Json.Linq.JToken.ToObject(Type objectType)
   at Newtonsoft.Json.Linq.JToken.ToObject[T]()
   at Sitecore.Xdb.Collection.Search.Solr.SolrDocuments.RebuildStatusDocument..ctor(DateTime lastUpdated, Int32 rebuildState, Double progressTotal, Double progressCurrentStage, Byte[] tokenBytes, JContainer rebuildcursors)
   at Void .ctor(System.DateTime, Int32, Double, Double, Byte[], Newtonsoft.Json.Linq.JContainer)(Object[] )
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObjectUsingCreatorWithParameters(JsonReader reader, JsonObjectContract contract, JsonProperty containerProperty, ObjectConstructor`1 creator, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateList(IList list, JsonReader reader, JsonArrayContract contract, JsonProperty containerProperty, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateList(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, Object existingValue, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.ResolvePropertyAndCreatorValues(JsonObjectContract contract, JsonProperty containerProperty, JsonReader reader, Type objectType)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObjectUsingCreatorWithParameters(JsonReader reader, JsonObjectContract contract, JsonProperty containerProperty, ObjectConstructor`1 creator, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.ResolvePropertyAndCreatorValues(JsonObjectContract contract, JsonProperty containerProperty, JsonReader reader, Type objectType)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObjectUsingCreatorWithParameters(JsonReader reader, JsonObjectContract contract, JsonProperty containerProperty, ObjectConstructor`1 creator, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
   at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
   at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonSerializerSettings settings)
   at Newtonsoft.Json.JsonConvert.DeserializeObject[T](String value, JsonSerializerSettings settings)
   at Sitecore.Xdb.Collection.Search.Solr.SolrResults.JsonResultsParser.ParseSearchResponse[TDoc](String solrResponse)
   at Sitecore.Xdb.Collection.Search.Solr.SolrRebuildStatusStorage.<ReadStatus>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Sitecore.Xdb.Collection.Indexing.Rebuild.IndexRebuildFlow.<TryToResumeRebuild>d__25.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Sitecore.Xdb.Collection.Indexing.Rebuild.IndexRebuildFlow.RebuildStatusWatcher.<>c__DisplayClass5_0.<<Start>g__RunRebuildFlowCycle|0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Sitecore.Xdb.Collection.Indexing.Rebuild.TaskExtension.<ExecuteWithInterval>d__0.MoveNext()

---

3/31 - Update

@x3mxray - This is a clone of an existing solution. I deleted all the documents while debugging this and have since done a rebuild. The xdb-index-token documents in both the main and rebuild indexes are correct (thanks for the script). I deleted all documents but those and kicked off a rebuild. I see the rebuild documents in the main index and the rebuild index is empty. When the rebuild completes, the main index has the state 5 & 6 rebuild documents and the rebuild index has state 1 - 5 rebuild documents and the errors start appearing.

---

4/1 - Update

The issue is that the config files for the XDB Solr indexes were not correct. After the config files were updated, the index rebuilt correctly and no more errors were generated.

We are taking this site over from other developers and this is for the local instance that I am setting up. The developers recently completed an upgrade from Sitecore 9.0 to Sitecore 9.3 and they used copies of the 9.0 indexes for 9.3 and a step must have been missed. It is an easy mistake to make.

@x3mxray - I am going to mark your answer as correct. If the config of the indexes had been correct, we would probably have run into the issue of them pointing at the wrong database, since this is a clone of an existing solution.

Answer 1

It looks like environment specific issue (especially if you clone solution to other environment).

First of all, try to check xdb-token value in your Sorl instance:

• Open your xdb index in Solr
• query for id:xdb-index-token and copy xdbtokenbytes_s field value
• run this powershell script (thanks to @richardszalay ConvertFrom-XdbIndexToken.ps1) and paste xdbtokenbytes_s value:

param(
    [string]$Token = (Read-Host "Enter xdbtokenbytes_s value")
)

$bytes = [Convert]::FromBase64String($Token)
$ms = [System.IO.MemoryStream]::new($bytes, 0, $bytes.Length)
$bf = [System.Runtime.Serialization.Formatters.Binary.BinaryFormatter]::new()
$dict = $bf.Deserialize($ms);

Write-Output $dict

You should see decrypted shard keys. Just be sure that they are pointing to your correct databases.

If:

• they are pointing to wrong databases
• id:xdb-index-token is missed in Solr
• xdbtokenbytes_s is empty

Then: Double check your environment configuration (config files). If your configs are correct then after XConnectSearchIndexer.exe -requestrebuild execution xdbtokenbytes_s will be populated with correct value for your environment.