Response time and resource usage increase during 4 hours

Question

In the last days we are experiencing some performance problems in our Sitecore.

Currently we are using Sitecore 9.0 as PaaS in Azure, in the last weeks we have added new sites on it. Right now we have 6-7 sites, each one having between 2500 and 7000 items.

In the last days we have noticed that just after 24h of a deploy, the response time of Sitecore increase suddenly from 200-300 ms to 20-40 seconds, during about 30-40 minutes, and after that start to decrease to 4-6 seconds returning to normal values (200-300ms) after 4 hours of starting the problem. It´s happenning every day, at the time of the last deploy.

Monitoring Sitecore we notice a big increase in CPU usage, in the number of threads for each instance, the write operations to azure redis cache increase a lot for first 5-10 minutes and when redis operations returns to normal values then increase a lot the DB operations during 4 hours aprox. After the 4 hours the CPU usage, number of threads and DB usage return to it´s usual values.

After suffer this problems during two days, we decided to add two more sitecore instance from 4 to 6 (PV2), and scale up Sql DB to 1600 DTU. The problem is mitigated but we still having the same problem with better time response, increasing to 2-3 seconds for few minutes, and 400-600 ms during 4 hours.

Looking at newrelic during the performance problem we have seen that the Sitecore Controller in the step AcquireRequestState is increasing from few ms to 20-40 seconds before scale out Sitecore. After scale-out it increases about 200-300 ms.

At Sitecore logs we don´t have seen anything unusual when the problem starts. A part from during all the day a lot warnings of cleaning AccessRequestCache , 30-50 each minutes, HttpModule being initialized about 20-30 per minute, and ThreadManagerInfo every 2 minutes approx.

We have been investigating but we don´t have a clue of what is happening. Any idea? It seems that Sitecore is doing something in background, but we don´t know what. The number of req/sec it´s in usual values just before start the problem.

Answer 1

We experienced a similar issue. Apparently Sitecore uses the session provider as sort of a temporary store that is then "processed" at a regular interval. This interval is controlled by the pollingInterval attribute on the session state provider configuration.

Every x number of seconds Sitecore will poll the session database for expired sessions. It will then foreach through all the found sessions and make an internal web request to the local IIS to "process" the session.

We found that this was exhausting IIS's request processing capability and normal page requests would start queuing up - which appears as a hung page to the end user.

We set the pollingInterval to 15, but other values may work better for you.

See https://kb.sitecore.net/articles/858026

In the Web.config and Sitecore.Analytics.Tracking.config files, increase the pollingInterval attribute value on both Private and Shared session state provider definitions from 2 to 60 seconds.

Increase the default polling interval for Sitecore Shared session state if Analytics tracking is enabled:

In the Sitecore.Analytics.Tracking.config file, find the timeoutBetweenLockAttempts parameter. Increase its value from 10 to 200 milliseconds. Note: This increases the waiting time between attempts to lock contact in the shared session state.

edit: Sitecore support personnel also suggested that we create a separate "processing" server and only do session/analytics processing there, but we were not able to try that.