Getting exception "Sync token is no longer valid for [Contacts] / [Interactions] table" with xConnect Search Indexer

Question

I had some trouble with xConnect at the beginning after I have installed Sitecore 9 Update 2. I had trouble with Sitecore xConnect Search Indexer Windows service particularly. It was a configuration issue at the end which caused the service not to run for a couple of days. Meanwhile, xConnect was saving data in the Contacts/Interactions table in SQL Server shard DBs.

When I fixed the issues with Search Indexer Windows Service, I am getting these exceptions and it's flooding my AppDomain log located under $path_to_your_xconnect_site$\App_data\jobs\continuous\IndexWorker\App_data\Logs.

Exception:

2018-08-09 10:13:07.158 +02:00 [Error] *** [GetContactsChanges], Line
27. Errno 50000: Sync token is no longer valid for [Contacts] table. System.Data.SqlClient.SqlException (0x80131904): *** [GetContactsChanges], Line 27. Errno 50000: Sync token is no longer valid for [Contacts] table.    at System.Data.SqlClient.SqlCommand.<>c.<ExecuteDbDataReaderAsync>b__174_0(Task`1 result)    at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke() at System.Threading.Tasks.Task.Execute()
--- End of stack trace from previous location where exception was thrown ---    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)    at Sitecore.Xdb.Sql.Common.Extensions.DbCommandExtensions.<>c__DisplayClass1_0.<<ExecuteReaderWithRetryAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)    at Sitecore.Xdb.Sql.Common.Extensions.DbCommandExtensions.<ExecuteReaderWithRetryAsync>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)    at Sitecore.Xdb.Sql.Common.Extensions.SqlCommandExtensions.<ExecuteReaderWithRetryAsync>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)    at Sitecore.Xdb.Sql.Common.Extensions.SqlCommandExtensions.<ExecuteReaderWithRetryAsync>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)    at Sitecore.Xdb.Collection.Data.SqlServer.Managers.SqlDataRecordsManager`2.<>c__DisplayClass75_0.<<ReadChanges>b__0>d.MoveNext() ClientConnectionId:0970f150-17f0-4f1a-b752-8974c0683f83 Error Number:50000,State:1,Class:16

Same exception but for [InteractionFacets], [ContactFacets] and [Interactions] is following this one with pretty much the same stack trace.

Has somebody encountered similar problems? What was the resolution?

Answer 1

The indexer can no longer figure out all the changes saved to xConnect while it was down. This is why rebuilding the index is the only option at this point.

———

Full context: if you are using SQL server for your collection database, the indexer uses SQL Server’s change tracking feature to keep up to date with any incoming changes.

SQL Server only keeps changes for a limited amount of time (configurable, I think default deployments have it at 5 days) called the retention period

Since your indexer was down longer than the retention period, then SQL Server will no longer returns all changes.

Answer 2

I ran into this a while ago but could not figure out how to fix it without a full reinstall.

However, recently @ezlateva and @jflarente figured out that you can fix this issue with an index rebuild of the xDB core. Note that this requires two cores: old and new. The steps required are specific to the search engine, but documentation has steps for Solr:

https://doc.sitecore.com/xp/en/developers/latest/sitecore-experience-platform/rebuild-the-xdb-index-in-solr.html

To invoke the rebuild request command:

1. Go to the server where the xConnect Search Indexer is running and open a command line.

2. Go to the xConnect Collection Search IndexWorker folder.

3. Run Sitecore.XConnectSearchIndexer.exe -rr (alternatively Sitecore.XConnectSearchIndexer.exe -requestrebuild). The command registers a small document in the live core signaling that the rebuild should be started. The xConnect Search Indexer will notice the document and start the rebuild process.

There are many more details in the docs, can't clone it all over, but hopefully this helps.

Answer 3

I got this error recently. We are hosting our Sitecore 9.2 instance on Azure PaaS and using Azure Cognitive Search.

When this exception comes, this usually indicates the sync token is no longer valid for the "Contacts" table. This means that the token used by xConnect to keep track of changes to the "Contacts" table has become invalid, and as a result, xConnect is unable to retrieve any changes to the table. This can be solved by rebuilding the xDB index in Solr as described here:

But for us, even index rebuilding is not able to finish. Then, we checked the Azure Search service and found out storage has reached out to the 99%. After we increase the scaled-up the Azure search service, this issue got resolved.

It's important to keep an eye on storage limits when using cloud services like Azure to avoid running into issues like this. Regularly monitoring and adjusting storage limits can help prevent unexpected errors and downtime.