Azure Web App Redis timeout exceptions when under load

Question

We are performance testing our Sitecore hosted in azure web apps (2 CD S3) which uses Premium Redis.

There are no error during normal UAT and SIT. However, on performance testing there are quiet a lot of errors similar to one below:

I've done below per this site https://gist.github.com/JonCole/db0e90bedeb3fc4823c2#diagnosing-redis-errors-on-the-client-side but still seeing these errors:

1. Increased the minThread to 200
2. Increase retry to 6000
3. Increase connection time out to 10000
4. Increase SyncTimeout to 5000

   
10696 2017:06:01 00:29:55 ERROR Cannot create tracker.
   Exception: System.TimeoutException
   Message: Timeout performing EVAL, inst: 86, mgr: Inactive, err: never, queue: 72, qu: 3, qs: 69, qc: 0, wr: 1, wq: 1, in: 65536, ar: 0, IOCP: (Busy=3,Free=997,Min=4,Max=1000), WORKER: (Busy=123,Free=32644,Min=200,Max=32767), clientName: RD0003FF8C5B59
   Source: StackExchange.Redis.StrongName
      at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor`1 processor, ServerEndPoint server)
      at StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor`1 processor, ServerEndPoint server)
      at StackExchange.Redis.RedisDatabase.ScriptEvaluate(String script, RedisKey[] keys, RedisValue[] values, CommandFlags flags)
      at Sitecore.SessionProvider.Redis.StackExchangeClientConnection.c__DisplayClass7.b__6()
      at Sitecore.SessionProvider.Redis.StackExchangeClientConnection.RetryForScriptNotFound(Func`1 redisOperation)
      at Sitecore.SessionProvider.Redis.StackExchangeClientConnection.RetryLogic(Func`1 redisOperation)
      at Sitecore.SessionProvider.Redis.RedisConnectionWrapper.TryTakeWriteLockAndGetData(String sessionId, DateTime lockTime, Object& lockId, ISessionStateItemCollection& data, Int32& sessionTimeout)
      at Sitecore.SessionProvider.Redis.RedisSessionStateProvider.GetItemFromSessionStore(Boolean isWriteLockRequired, HttpContext context, String id, Boolean& locked, TimeSpan& lockAge, Object& lockId, SessionStateActions& actions)
      at Sitecore.SessionProvider.Redis.RedisSessionStateProvider.GetItemExclusive(HttpContext context, String id, Boolean& locked, TimeSpan& lockAge, Object& lockId, SessionStateActions& actions)
      at Sitecore.Analytics.Tracking.SharedSessionState.SharedSessionStateManager.LockAndLoadContact(Guid contactId)
      at Sitecore.Analytics.Tracking.ContactManager.TryLoadContact(Guid contactId, Int32 lockDurationMinutes, Boolean exclusive)
      at Sitecore.Analytics.Tracking.ContactManager.TryLoadContact(Guid contactId, Boolean exclusive)
      at Sitecore.Analytics.Pipelines.EnsureSessionContext.LoadContact.Process(InitializeTrackerArgs args)
      at (Object , Object[] )
      at Sitecore.Pipelines.CorePipeline.Run(PipelineArgs args)
      at Sitecore.Pipelines.DefaultCorePipelineManager.Run(String pipelineName, PipelineArgs args, String pipelineDomain)
      at Sitecore.Analytics.DefaultTracker.EnsureSessionContext()
      at Sitecore.Analytics.Pipelines.CreateTracker.GetTracker.Process(CreateTrackerArgs args)
      at (Object , Object[] )
      at Sitecore.Pipelines.CorePipeline.Run(PipelineArgs args)
      at Sitecore.Pipelines.DefaultCorePipelineManager.Run(String pipelineName, PipelineArgs args, String pipelineDomain)
      at Sitecore.Analytics.Tracker.Initialize()

Any help will be good.

Here's our setting for shared session:

<?xml version="1.0"?>
<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/" xmlns:set="http://www.sitecore.net/xmlconfig/set/">
  <sitecore>
    <tracking>
      <sharedSessionState>
        <patch:attribute name="defaultProvider">redis</patch:attribute>
        <providers>
          <add name="redis" type=" Sitecore.SessionProvider.Redis.RedisSessionStateProvider, Sitecore.SessionProvider.Redis" connectionString="redis.sessions"
            pollingInterval="2" applicationName="shared" patch:instead="add[@name='InProc']"/>
        </providers>
      </sharedSessionState>
    </tracking>
  </sitecore>
</configuration>

Thanks.

Answer 1

We managed to solve this issue with the help of Sitecore and Microsoft:

Solution:

1. Increase the min worker thread pool to xxx (where xxx should be greater than busy value on log)

2. Increase PollingTime to 30.

3. If above doesn't help upgrade the redis.stackexcahange to latest

4. And increase sync timeout 5k, connection time 10k and retry timeout 6k (should be greater than sync timeout)

Answer 2

This blog post covers a great explanation from Sitecore about what's going on when you see exceptions like this and how to adjust config to fix the problem.

The general approach as recommended by Sitecore seems to be first to apply support patch Sitecore.Support.210408 and config changes from here. A note that (in my opinion) they don't supply very good default settings for the thread limits. From my testing on a production site I'd recommend as follows (of course every solution is different so you'll still need to fine-tune over time).

<MaxWorkerThreadsPerCore>100</MaxWorkerThreadsPerCore>                        
<MinWorkerThreadsPerCore>50</MinWorkerThreadsPerCore>
<MaxCompletionPortThreadsPerCore>100</MaxCompletionPortThreadsPerCore>
<MinCompletionPortThreadsPerCore>50</MinCompletionPortThreadsPerCore>

Next up is to go through this KB article and consider the options recommended there - the main one being to decide whether to add another redis instance and split the private and shared session state.

Session Weight

Something that isn't really mentioned is session weight. Redis, being a key-value store, performs best when storing a lot of very small light weight values. From my experience, if your private session is holding more than about 1MB per session then you're going to have bad performance under load that will eventually lead to site degradation.

So take a look at your code and see if you can optimize session weight. Do you really need to store an entire member's profile in session once they log in, or can you just track some important values and get the rest of it from another data store when required?

Update

Further along the lines of Session Weight - a great post from Bartłomiej Mucha confirms that Redis performance scales linearly with cache size, and since the session is stored in the cache then this is a problem. So scaling up Redis to the next tier will help a lot if you diagnose it as hitting the request per second limit for that cache size.

Answer 3

I had this issue on 9.1 Update 1, none of the above worked, but when i added Hotfix 355701 and scaled up my App Service up from P1V2 to P3V2 and it went away.

Answer 4

As @MarkGibbons mentions in his answer, session weight can be very heavy if proper maintenance and configuration is not in place. Recently, we faced a situation where the average session size was over 12MB and this was the leading cause for why we experienced these timeouts.

Review Session Data

• Install stunnel
• Install Redis CLI

Once connected to Redis, issue the commands:

• slowlog get 10
• Find a command with a large size, then issue hgetall {....._Data} (Key is pulled from slowlog output)

This will help identify why the session object is so large. It can be either private or shared session (it was both in our case).

Reduce Session Size

• Remove old xDB data
  • This will reduce the shared session size
  • You can use the Analytics Database Manager module to help truncate data
• Reduce MaxPageIndexThreshold setting on SetDummySession processor
  • This will reduce the private session size
• Increase the Analytics.LongRunningRequest setting
  • When a long-running request occurs, it is logged to the session. Increasing the value reduces the likelihood of it being logged.
• Change Analytics.LogLevel to None
  • Exceptions that occur during a user session are logged to session. Stack traces can eat up a considerable number of bytes. Changing to None will prevent them from being written to session.
• Add a custom processor to <ensureSessionContext> pipeline
  • This is the same pipeline where the SetDummySession processor limits the session size. You can add a custom processor here to limit based on your needs. E.g. limit the number of page events per page to avoid spamming.
• Enable compression
  • This is one of the best places to begin. Enabling compression is very simple and can yield incredible results. Sitecore warns about an increase in CPU utilization, but this has not yet been witnessed. Find more at my link below.

You may do this for private or shared session...

<add name="redis" compression="true" ... />

More info on my blog

Diagnose and trim data from session

Investigation into compressing data sent to Redis