In the last days we are experiencing some performance problems in our Sitecore.
Currently we are using Sitecore 9.0 as PaaS in Azure, in the last weeks we have added new sites on it. Right now we have 6-7 sites, each one having between 2500 and 7000 items.
In the last days we have noticed that just after 24h of a deploy, the response time of Sitecore increase suddenly from 200-300 ms to 20-40 seconds, during about 30-40 minutes, and after that start to decrease to 4-6 seconds returning to normal values (200-300ms) after 4 hours of starting the problem. It´s happenning every day, at the time of the last deploy.
Monitoring Sitecore we notice a big increase in CPU usage, in the number of threads for each instance, the write operations to azure redis cache increase a lot for first 5-10 minutes and when redis operations returns to normal values then increase a lot the DB operations during 4 hours aprox. After the 4 hours the CPU usage, number of threads and DB usage return to it´s usual values.
After suffer this problems during two days, we decided to add two more sitecore instance from 4 to 6 (PV2), and scale up Sql DB to 1600 DTU. The problem is mitigated but we still having the same problem with better time response, increasing to 2-3 seconds for few minutes, and 400-600 ms during 4 hours.
Looking at newrelic during the performance problem we have seen that the Sitecore Controller in the step AcquireRequestState is increasing from few ms to 20-40 seconds before scale out Sitecore. After scale-out it increases about 200-300 ms.
At Sitecore logs we don´t have seen anything unusual when the problem starts. A part from during all the day a lot warnings of cleaning AccessRequestCache , 30-50 each minutes, HttpModule being initialized about 20-30 per minute, and ThreadManagerInfo every 2 minutes approx.
We have been investigating but we don´t have a clue of what is happening. Any idea? It seems that Sitecore is doing something in background, but we don´t know what. The number of req/sec it´s in usual values just before start the problem.