Performance issue on page load Sitecore.Analytics BLOCKED_TIME

Question

Background of Infra/Software:

Sitecore Product Version: 9.2

Sitecore XP Scaled topology used.

Infrastructure Type: Azure App Service / PaaS inside Isolated ASE

Issue:

Experiencing slowness in Page load, sometimes it is taking more than a minute. On checking the Profiler trace we found that the Sitecore analytics process is getting blocked while creating a tracker object.

Stack trace

     OTHER <<ntdll!RtlUserThreadStart>>
   Sitecore.Mvc!Sitecore.Mvc.Routing.RouteHttpHandler.BeginProcessRequest
   Sitecore.Mvc!Sitecore.Mvc.Pipelines.PipelineService.RunPipeline
   Sitecore.Kernel!Sitecore.Pipelines.DefaultCorePipelineManager.Run
   Sitecore.Kernel!Sitecore.Pipelines.DefaultCorePipelineManager.Run
   Sitecore.Kernel!Sitecore.Pipelines.CorePipeline.Run
   Sitecore.Mvc.Analytics!dynamicClass.
   Sitecore.Mvc.Analytics!Sitecore.Mvc.Analytics.Pipelines.MvcEvents.RequestBegin.StartTracking.Process
   Sitecore.Analytics.XConnect!Sitecore.Analytics.XConnect.Diagnostics.PerformanceCounters.OperationPerformanceMonitorExtensions.Monitor
   Sitecore.Analytics.XConnect!Sitecore.Analytics.XConnect.Diagnostics.PerformanceCounters.OperationPerformanceMonitorExtensions+<>c__DisplayClass1...
   Sitecore.Kernel!Sitecore.Pipelines.CorePipeline.Run
   Sitecore.Analytics!dynamicClass.
   Sitecore.Analytics!Sitecore.Analytics.Tracker.Initialize
   Sitecore.Kernel!Sitecore.Pipelines.DefaultCorePipelineManager.Run
   Sitecore.Kernel!Sitecore.Pipelines.DefaultCorePipelineManager.Run
   Sitecore.Kernel!Sitecore.Pipelines.CorePipeline.Run
   Sitecore.Analytics!dynamicClass.
   Sitecore.Analytics!Sitecore.Analytics.Pipelines.CreateTracker.GetTracker.Process
   Sitecore.Analytics!Sitecore.Analytics.DefaultTracker.EnsureSessionContext
   Sitecore.Analytics.XConnect!Sitecore.Analytics.XConnect.Diagnostics.PerformanceCounters.OperationPerformanceMonitorExtensions.Monitor
   Sitecore.Analytics.XConnect!Sitecore.Analytics.XConnect.Diagnostics.PerformanceCounters.OperationPerformanceMonitorExtensions+<>c__DisplayClass1...
   Sitecore.Analytics!Sitecore.Analytics.Pipelines.EnsureSessionContext.EnsureSessionContextPipeline+<>c__DisplayClass4_0.<Run>b__0
   Sitecore.Kernel!Sitecore.Pipelines.DefaultCorePipelineManager.Run
   Sitecore.Kernel!Sitecore.Pipelines.DefaultCorePipelineManager.Run
   Sitecore.Kernel!Sitecore.Pipelines.CorePipeline.Run
   Sitecore.Analytics!dynamicClass.
   Sitecore.Analytics!Sitecore.Analytics.Pipelines.EnsureSessionContext.LoadContact.Process
   Sitecore.Analytics!Sitecore.Analytics.Tracking.ContactManager.LoadContact
   Sitecore.Analytics!Sitecore.Analytics.Data.ContactRepository.LoadContact
   Sitecore.Analytics.XConnect!Sitecore.Analytics.XConnect.Diagnostics.PerformanceCounters.OperationPerformanceMonitorExtensions.Monitor
   Sitecore.Analytics.XConnect!Sitecore.Analytics.XConnect.DataAccess.XConnectDataAdapterProvider+<>c__DisplayClass15_0.<LoadContact>b__0
   Sitecore.Analytics.XConnect!Sitecore.Analytics.XConnect.DataAccess.XConnectDataAdapterProvider.ExecuteWithExceptionHandling
   Sitecore.XConnect.Client.Configuration!Sitecore.XConnect.Client.Configuration.SitecoreXConnectClientConfiguration.GetClient
   Sitecore.Kernel!Sitecore.Configuration.DefaultFactory.CreateObject
   Sitecore.Kernel!Sitecore.Configuration.DefaultFactory.CreateObject
   Sitecore.Kernel!Sitecore.Configuration.DefaultFactory.CreateObject
   Sitecore.XConnect.Client.Configuration!Sitecore.XConnect.Client.Configuration.SitecoreXConnectClientConfiguration.Initialize
   Sitecore.XConnect.Client!Sitecore.XConnect.Client.XConnectSynchronousExtensions.SuspendContextLock
   Sitecore.Xdb.Common.Web!Sitecore.Xdb.Common.Web.Synchronous.SynchronousExtensions.SuspendContextLock
   Sitecore.Xdb.Common.Web!Sitecore.Xdb.Common.Web.Synchronous.CurrentThreadTaskScheduler.ProcessTaskQueue
   OTHER <<System!System.Collections.Concurrent.BlockingCollection`1+<GetConsumingEnumerable>d__68[System.__Canon].MoveNext
   BLOCKED_TIME
OTHER <<clr!JITutil_MonTryEnter>>
   BLOCKED_TIME
LAST_BLOCK (Last blocking operation in trace)
   BLOCKED_TIME

Answer 1

Just for everyone's benefit: Finally found the root cause of the connectivity issue between app services.

As part Azure DevOps migration to new server a new Route to the Route table was added which was causing the traffic from inside the ASE to be routed out causing communication outage.

This route was deleted and so the connectivity was restored & then Sitecore Analytics was enabled back which was temporarily disabled earlier.

For more details check out my blog : https://ratishkr81.blogspot.com/2021/12/sitecore-performance-issue-on-page-load.html