10

We are performance testing our Sitecore hosted in azure web apps (2 CD S3) which uses Premium Redis.

There are no error during normal UAT and SIT. However, on performance testing there are quiet a lot of errors similar to one below:

I've done below per this site https://gist.github.com/JonCole/db0e90bedeb3fc4823c2#diagnosing-redis-errors-on-the-client-side but still seeing these errors:

  1. Increased the minThread to 200
  2. Increase retry to 6000
  3. Increase connection time out to 10000
  4. Increase SyncTimeout to 5000
   
10696 2017:06:01 00:29:55 ERROR Cannot create tracker.
   Exception: System.TimeoutException
   Message: Timeout performing EVAL, inst: 86, mgr: Inactive, err: never, queue: 72, qu: 3, qs: 69, qc: 0, wr: 1, wq: 1, in: 65536, ar: 0, IOCP: (Busy=3,Free=997,Min=4,Max=1000), WORKER: (Busy=123,Free=32644,Min=200,Max=32767), clientName: RD0003FF8C5B59
   Source: StackExchange.Redis.StrongName
      at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor`1 processor, ServerEndPoint server)
      at StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor`1 processor, ServerEndPoint server)
      at StackExchange.Redis.RedisDatabase.ScriptEvaluate(String script, RedisKey[] keys, RedisValue[] values, CommandFlags flags)
      at Sitecore.SessionProvider.Redis.StackExchangeClientConnection.c__DisplayClass7.b__6()
      at Sitecore.SessionProvider.Redis.StackExchangeClientConnection.RetryForScriptNotFound(Func`1 redisOperation)
      at Sitecore.SessionProvider.Redis.StackExchangeClientConnection.RetryLogic(Func`1 redisOperation)
      at Sitecore.SessionProvider.Redis.RedisConnectionWrapper.TryTakeWriteLockAndGetData(String sessionId, DateTime lockTime, Object& lockId, ISessionStateItemCollection& data, Int32& sessionTimeout)
      at Sitecore.SessionProvider.Redis.RedisSessionStateProvider.GetItemFromSessionStore(Boolean isWriteLockRequired, HttpContext context, String id, Boolean& locked, TimeSpan& lockAge, Object& lockId, SessionStateActions& actions)
      at Sitecore.SessionProvider.Redis.RedisSessionStateProvider.GetItemExclusive(HttpContext context, String id, Boolean& locked, TimeSpan& lockAge, Object& lockId, SessionStateActions& actions)
      at Sitecore.Analytics.Tracking.SharedSessionState.SharedSessionStateManager.LockAndLoadContact(Guid contactId)
      at Sitecore.Analytics.Tracking.ContactManager.TryLoadContact(Guid contactId, Int32 lockDurationMinutes, Boolean exclusive)
      at Sitecore.Analytics.Tracking.ContactManager.TryLoadContact(Guid contactId, Boolean exclusive)
      at Sitecore.Analytics.Pipelines.EnsureSessionContext.LoadContact.Process(InitializeTrackerArgs args)
      at (Object , Object[] )
      at Sitecore.Pipelines.CorePipeline.Run(PipelineArgs args)
      at Sitecore.Pipelines.DefaultCorePipelineManager.Run(String pipelineName, PipelineArgs args, String pipelineDomain)
      at Sitecore.Analytics.DefaultTracker.EnsureSessionContext()
      at Sitecore.Analytics.Pipelines.CreateTracker.GetTracker.Process(CreateTrackerArgs args)
      at (Object , Object[] )
      at Sitecore.Pipelines.CorePipeline.Run(PipelineArgs args)
      at Sitecore.Pipelines.DefaultCorePipelineManager.Run(String pipelineName, PipelineArgs args, String pipelineDomain)
      at Sitecore.Analytics.Tracker.Initialize()

Any help will be good.

Here's our setting for shared session:

<?xml version="1.0"?>
<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/" xmlns:set="http://www.sitecore.net/xmlconfig/set/">
  <sitecore>
    <tracking>
      <sharedSessionState>
        <patch:attribute name="defaultProvider">redis</patch:attribute>
        <providers>
          <add name="redis" type=" Sitecore.SessionProvider.Redis.RedisSessionStateProvider, Sitecore.SessionProvider.Redis" connectionString="redis.sessions"
            pollingInterval="2" applicationName="shared" patch:instead="add[@name='InProc']"/>
        </providers>
      </sharedSessionState>
    </tracking>
  </sitecore>
</configuration>

Thanks.

  • Are Redis and your CD instances in the same Azure region? Can you provide the full sharedSessionState configuration node from showconfig.aspx? – maz Jun 1 '17 at 13:51
  • @maz see updated details. – Nil Pun Jun 2 '17 at 1:07
  • Same issue here. Azure Redis and CD App Service instances are both in North Europe. sharedSessionState configuration is: <sharedSessionState defaultProvider="InProc" patch:source="Sitecore.Analytics.Tracking.config"> <providers> <clear /> <add name="InProc" type="System.Web.SessionState.InProcSessionStateStore" /> </providers> <manager type="Sitecore.Analytics.Tracking.SharedSessionState.SharedSessionStateManager, Sitecore.Analytics"> <param ref="tracking/sharedSessionState/config" desc="configuration" /> </manager> <config type="Sitecore.Analytics.Tracking.SharedSessionState.SharedSessionStat – António Ribeiro Jul 17 '17 at 11:09
  • Hi Antonio and welcome to Sitecore Stack Exchange :-). Your answer is more a comment, not an actual answer to the question. I will convert it to a comment for you, since you need to build up a bit of reputation on the site before you can comment directly. – Mark Cassidy Jul 17 '17 at 11:33
  • We are facing similar issue . can you please provide more details about the Resolution steps. – Balaji Balasubramanian Feb 9 '18 at 14:44
5

We managed to solve this issue with the help of Sitecore and Microsoft:

Solution:

  1. Increase the min worker thread pool to xxx (where xxx should be greater than busy value on log)

  2. Increase PollingTime to 30.

  3. If above doesn't help upgrade the redis.stackexcahange to latest

  4. And increase sync timeout 5k, connection time 10k and retry timeout 6k (should be greater than sync timeout)

2

This blog post covers a great explanation from Sitecore about what's going on when you see exceptions like this and how to adjust config to fix the problem.

The general approach as recommended by Sitecore seems to be first to apply support patch Sitecore.Support.210408 and config changes from here. A note that (in my opinion) they don't supply very good default settings for the thread limits. From my testing on a production site I'd recommend as follows (of course every solution is different so you'll still need to fine-tune over time).

<MaxWorkerThreadsPerCore>100</MaxWorkerThreadsPerCore>                        
<MinWorkerThreadsPerCore>50</MinWorkerThreadsPerCore>
<MaxCompletionPortThreadsPerCore>100</MaxCompletionPortThreadsPerCore>
<MinCompletionPortThreadsPerCore>50</MinCompletionPortThreadsPerCore>

Next up is to go through this KB article and consider the options recommended there - the main one being to decide whether to add another redis instance and split the private and shared session state.

Session Weight

Something that isn't really mentioned is session weight. Redis, being a key-value store, performs best when storing a lot of very small light weight values. From my experience, if your private session is holding more than about 1MB per session then you're going to have bad performance under load that will eventually lead to site degradation.

So take a look at your code and see if you can optimize session weight. Do you really need to store an entire member's profile in session once they log in, or can you just track some important values and get the rest of it from another data store when required?

  • do you have recommendations for how to tune or calculate the thread limits for an environment? – Mike J Apr 12 at 20:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.