Experience analytics throwing Sitecore.Xdb.Common.Web.ConnectionTimeoutException

Question

We are getting below exceptions while trying to load graphs on the experience analytics dashboard. The dashboard timer keep on spinning and after few seconds we get the well known bunch of exceptions on the UI:

The OnlineInteractions graph cannot be displayed due to a server error. Contact you system administrator.

Whereas in CM logs we see Sitecore.XConnect.XdbCollectionUnavailableException and Sitecore.Xdb.Common.Web.ConnectionTimeoutException, please see the stack trace below.

While debugging we already had a look at this question and make sure the collectiouser have a required rights on Shard databases. Though we don't see any post installation steps in 9.0.2 installation guide to execute such script, hence we assume on v9.0.2 it is being taken care while installation itself.

What could be other possible causes for this Timeout exception?

Exception: Sitecore.XConnect.XdbCollectionUnavailableException
Message: An error occurred while sending the request.
Source: Sitecore.Xdb.Common.Web
   at Sitecore.Xdb.Common.Web.Synchronous.SynchronousExtensions.SuspendContextLock[TResult](Func`1 taskFactory)
   at Sitecore.XConnect.Client.XConnectSynchronousExtensions.SuspendContextLock(Func`1 taskFactory)
   at Sitecore.XConnect.Client.Configuration.SitecoreXConnectClientConfiguration.Initialize(XmlNode configNode)
   at Sitecore.Configuration.DefaultFactory.CreateObject(XmlNode configNode, String[] parameters, Boolean assert, IFactoryHelper helper)
   at Sitecore.Configuration.DefaultFactory.CreateObject(XmlNode configNode, String[] parameters, Boolean assert)
   at Sitecore.Configuration.DefaultFactory.CreateObject(String configPath, String[] parameters, Boolean assert)
   at Sitecore.XConnect.Client.Configuration.SitecoreXConnectClientConfiguration.GetClient(String clientConfigPath)
   at Sitecore.PathAnalyzer.Processing.Agents.TreeAggregatorAgent.Execute()
   at Sitecore.Analytics.Core.BackgroundService.Run()

Exception: Sitecore.Xdb.Common.Web.ConnectionTimeoutException
Message: A task was canceled.
Source: Sitecore.Xdb.Common.Web
   at Sitecore.Xdb.Common.Web.CommonWebApiClient`1.<ExecuteAsync>d__37.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Sitecore.Xdb.Common.Web.CommonWebApiClient`1.<ExecuteGetAsync>d__32.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Sitecore.XConnect.Client.WebApi.ConfigurationWebApiClient.<Refresh>d__4.MoveNext()

Nested Exception

Exception: System.Threading.Tasks.TaskCanceledException
Message: A task was canceled.
Source: mscorlib
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Sitecore.Xdb.Common.Web.CommonWebApiClient`1.<ExecuteAsync>d__37.MoveNext()

Logs in xDB Reference Data Service

2019-03-12 13:27:52.417 -04:00 [Information] XConnect Test Host Application End, Machine: "VM-Name", Site: "stg9.xdb-reference-data.mydomain.com", AppId: "/LM/W3SVC/6/ROOT"
2019-03-12 13:28:31.531 -04:00 [Information] XConnect Test Host Application Start, Machine: "VM-Name", Site: "stg9.xdb-reference-data.mydomain.com", AppId: "/LM/W3SVC/6/ROOT"
2019-03-12 13:28:32.422 -04:00 [Information] Certificate Validation Filter Enabled, Thumbprint: mythumbprint
2019-03-12 13:28:32.422 -04:00 [Information] SSL Validation Filter Enabled

Browser console logs on Experience Dashboard:

Answer 1

Experience Analytics relies on both the Reporting Server of Sitecore and the Reference Data service of xConnect.

More often then not, this issue is because of an error occuring in the Reference Data service (which also relies on the Collection Search service)

To troubleshoot, I would examine the logs from the xConnect Reference Data service to see if there is a .NET error.

You may also have to look at the IIS access logs to see the request coming in from the Sitecore server and look at the result code.

Expect a 200 if everything is fine.

Expect a 403 if there are most likely certificate errors.

Expect a 500 if there are .NET errors.

Trying hard not to make this a link only answer as well as trying not to duplicate an answer already given, you may want to Validate Your Install of Sitecore 9.0.2 using this helpful guidance.