xDB Tracker Timeout with certain contacts only

Question

Interesting behavior here with Sitecore XP 9.3, any ideas are very welcomed.

What happens

Only for specific contacts, anything related to Tracker reading (Eg: IdentifyAs, LoadContact, etc) is dying with a timeout, but only on CDs, while it normally works on the CM (We use SQL shared sessions on the CDs)

Eg:

Tracker.Current.Session.IdentifyAs(source, value.ToString());

Error: xDB unavailable - StackTrace: at Sitecore.Analytics.XConnect.DataAccess.XConnectDataAdapterProvider.ExecuteWithExceptionHandling[T](Func2 func) at Sitecore.Analytics.XConnect.DataAccess.XConnectDataAdapterProvider.<>c__DisplayClass17_0.b__0() at Sitecore.Analytics.XConnect.Diagnostics.PerformanceCounters.OperationPerformanceMonitorExtensions.Monitor[T](OperationPerformanceMonitorBase monitor, Func1 operation) at Sitecore.Analytics.Data.ContactRepository.LoadContact(String source, String identifier) at Sitecore.Analytics.Tracking.ContactManager.LoadContact(String source, String identifier) at Sitecore.Analytics.Tracking.StandardSession.GetContactByIdentifier(String source, String identifier, ContactManager contactManager, Action`1 validateIdentifier) at Sitecore.Analytics.Tracking.StandardSession.GetContactByIdentifier(String source, String identifier, ContactManager contactManager) at Sitecore.Analytics.Tracking.StandardSession.IdentifyAs(String source, String knownIdentifier) at ASP.contactidentifyas_aspx.__Render__control1(HtmlTextWriter __w, Control parameterContainer)

Error: Operation was cancelled by timeout

Now comes the interesting part:

1. if I increase the Tracker Timeout to a high number (say 2 minutes), and run the same IdentifyAs code, instead of timeout, the identification will occur normally
2. After that, even if I decrease the timeout back to 5 seconds, this contact will not give a timeout again! And the IdentifyAs method runs pretty fast, by the way.

Looks like something large was stuck for this contact, and the IdentifyAs method when successfully completed is cleaning the contact.

Please let me know your ideas on how to attack this problem.

Answer 1

For scenarios such as this, if you are running a load balancer, it is sometimes worth experimenting with toggling sticky sessions in order to isolate variables. However, since you are using shared SQL sessions, I don’t believe that would be helpful in this particular case.

Otherwise, such timeouts can indicate that large data payloads are at play. Do you have any untracked pages set in your configuration? If not, perhaps you should. If your site makes many XHR calls, that can add up quickly and clog up the pipes. More reading:

https://doc.sitecore.com/xp/en/developers/90/sitecore-experience-platform/web-tracking.html#untracked-pages_body

I also have a blog post about how to test issues like this more easily:

https://sitecore.marcelgruber.ca/posts/xconnect-rapid-interaction-spoofing-contact-identification

Answer 2

This could be explained if the contact being identified has a large number of interactions. Then when that contact is loaded the volume of data being called via API takes time to load.

To identify contacts like this try running the following:

SELECT TOP (100) ContactId, COUNT(ContactId) as Count
FROM [xdb_collection].[Interactions]
GROUP BY ContactId
ORDER BY Count DESC

Then analyse the data using below:

SELECT TOP (100) *
    FROM [xdb_collection].[Interactions]
    WHERE ContactId ='xxx'

More info here: https://support.sitecore.com/kb?id=kb_article_view&sysparm_article=KB0417184

Answer 3

Seems the connectivity between your CD servers and the xConnect server was not stable. Maybe when a website was running and sending the information to xConnect from which CD servers didn't receive the response in a timely manner.

It doesn't mean the xConnect getting any problems or errors.

If your xConnect is hosted under IIS in a separate application, you can try to look at its App Pool settings. Try to change the Start Mode to AlwaysRunning and Idle-timeout Action to Suspend. This can prevent IIS from terminating the xConnect when it reaches the Idle-timeout (20 minutes as default).