10

We are performance testing our Sitecore hosted in azure web apps (2 CD S3) which uses Premium Redis.

There are no error during normal UAT and SIT. However, on performance testing there are quiet a lot of errors similar to one below:

I've done below per this site https://gist.github.com/JonCole/db0e90bedeb3fc4823c2#diagnosing-redis-errors-on-the-client-side but still seeing these errors:

  1. Increased the minThread to 200
  2. Increase retry to 6000
  3. Increase connection time out to 10000
  4. Increase SyncTimeout to 5000
   
10696 2017:06:01 00:29:55 ERROR Cannot create tracker.
   Exception: System.TimeoutException
   Message: Timeout performing EVAL, inst: 86, mgr: Inactive, err: never, queue: 72, qu: 3, qs: 69, qc: 0, wr: 1, wq: 1, in: 65536, ar: 0, IOCP: (Busy=3,Free=997,Min=4,Max=1000), WORKER: (Busy=123,Free=32644,Min=200,Max=32767), clientName: RD0003FF8C5B59
   Source: StackExchange.Redis.StrongName
      at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor`1 processor, ServerEndPoint server)
      at StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor`1 processor, ServerEndPoint server)
      at StackExchange.Redis.RedisDatabase.ScriptEvaluate(String script, RedisKey[] keys, RedisValue[] values, CommandFlags flags)
      at Sitecore.SessionProvider.Redis.StackExchangeClientConnection.c__DisplayClass7.b__6()
      at Sitecore.SessionProvider.Redis.StackExchangeClientConnection.RetryForScriptNotFound(Func`1 redisOperation)
      at Sitecore.SessionProvider.Redis.StackExchangeClientConnection.RetryLogic(Func`1 redisOperation)
      at Sitecore.SessionProvider.Redis.RedisConnectionWrapper.TryTakeWriteLockAndGetData(String sessionId, DateTime lockTime, Object& lockId, ISessionStateItemCollection& data, Int32& sessionTimeout)
      at Sitecore.SessionProvider.Redis.RedisSessionStateProvider.GetItemFromSessionStore(Boolean isWriteLockRequired, HttpContext context, String id, Boolean& locked, TimeSpan& lockAge, Object& lockId, SessionStateActions& actions)
      at Sitecore.SessionProvider.Redis.RedisSessionStateProvider.GetItemExclusive(HttpContext context, String id, Boolean& locked, TimeSpan& lockAge, Object& lockId, SessionStateActions& actions)
      at Sitecore.Analytics.Tracking.SharedSessionState.SharedSessionStateManager.LockAndLoadContact(Guid contactId)
      at Sitecore.Analytics.Tracking.ContactManager.TryLoadContact(Guid contactId, Int32 lockDurationMinutes, Boolean exclusive)
      at Sitecore.Analytics.Tracking.ContactManager.TryLoadContact(Guid contactId, Boolean exclusive)
      at Sitecore.Analytics.Pipelines.EnsureSessionContext.LoadContact.Process(InitializeTrackerArgs args)
      at (Object , Object[] )
      at Sitecore.Pipelines.CorePipeline.Run(PipelineArgs args)
      at Sitecore.Pipelines.DefaultCorePipelineManager.Run(String pipelineName, PipelineArgs args, String pipelineDomain)
      at Sitecore.Analytics.DefaultTracker.EnsureSessionContext()
      at Sitecore.Analytics.Pipelines.CreateTracker.GetTracker.Process(CreateTrackerArgs args)
      at (Object , Object[] )
      at Sitecore.Pipelines.CorePipeline.Run(PipelineArgs args)
      at Sitecore.Pipelines.DefaultCorePipelineManager.Run(String pipelineName, PipelineArgs args, String pipelineDomain)
      at Sitecore.Analytics.Tracker.Initialize()

Any help will be good.

Here's our setting for shared session:

<?xml version="1.0"?>
<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/" xmlns:set="http://www.sitecore.net/xmlconfig/set/">
  <sitecore>
    <tracking>
      <sharedSessionState>
        <patch:attribute name="defaultProvider">redis</patch:attribute>
        <providers>
          <add name="redis" type=" Sitecore.SessionProvider.Redis.RedisSessionStateProvider, Sitecore.SessionProvider.Redis" connectionString="redis.sessions"
            pollingInterval="2" applicationName="shared" patch:instead="add[@name='InProc']"/>
        </providers>
      </sharedSessionState>
    </tracking>
  </sitecore>
</configuration>

Thanks.

  • Are Redis and your CD instances in the same Azure region? Can you provide the full sharedSessionState configuration node from showconfig.aspx? – maz Jun 1 '17 at 13:51
  • @maz see updated details. – Nil Pun Jun 2 '17 at 1:07
  • Same issue here. Azure Redis and CD App Service instances are both in North Europe. sharedSessionState configuration is: <sharedSessionState defaultProvider="InProc" patch:source="Sitecore.Analytics.Tracking.config"> <providers> <clear /> <add name="InProc" type="System.Web.SessionState.InProcSessionStateStore" /> </providers> <manager type="Sitecore.Analytics.Tracking.SharedSessionState.SharedSessionStateManager, Sitecore.Analytics"> <param ref="tracking/sharedSessionState/config" desc="configuration" /> </manager> <config type="Sitecore.Analytics.Tracking.SharedSessionState.SharedSessionStat – António Ribeiro Jul 17 '17 at 11:09
  • Hi Antonio and welcome to Sitecore Stack Exchange :-). Your answer is more a comment, not an actual answer to the question. I will convert it to a comment for you, since you need to build up a bit of reputation on the site before you can comment directly. – Mark Cassidy Jul 17 '17 at 11:33
  • We are facing similar issue . can you please provide more details about the Resolution steps. – Balaji Balasubramanian Feb 9 '18 at 14:44
5

We managed to solve this issue with the help of Sitecore and Microsoft:

Solution:

  1. Increase the min worker thread pool to xxx (where xxx should be greater than busy value on log)

  2. Increase PollingTime to 30.

  3. If above doesn't help upgrade the redis.stackexcahange to latest

  4. And increase sync timeout 5k, connection time 10k and retry timeout 6k (should be greater than sync timeout)

| improve this answer | |
2

This blog post covers a great explanation from Sitecore about what's going on when you see exceptions like this and how to adjust config to fix the problem.

The general approach as recommended by Sitecore seems to be first to apply support patch Sitecore.Support.210408 and config changes from here. A note that (in my opinion) they don't supply very good default settings for the thread limits. From my testing on a production site I'd recommend as follows (of course every solution is different so you'll still need to fine-tune over time).

<MaxWorkerThreadsPerCore>100</MaxWorkerThreadsPerCore>                        
<MinWorkerThreadsPerCore>50</MinWorkerThreadsPerCore>
<MaxCompletionPortThreadsPerCore>100</MaxCompletionPortThreadsPerCore>
<MinCompletionPortThreadsPerCore>50</MinCompletionPortThreadsPerCore>

Next up is to go through this KB article and consider the options recommended there - the main one being to decide whether to add another redis instance and split the private and shared session state.

Session Weight

Something that isn't really mentioned is session weight. Redis, being a key-value store, performs best when storing a lot of very small light weight values. From my experience, if your private session is holding more than about 1MB per session then you're going to have bad performance under load that will eventually lead to site degradation.

So take a look at your code and see if you can optimize session weight. Do you really need to store an entire member's profile in session once they log in, or can you just track some important values and get the rest of it from another data store when required?

Update

Further along the lines of Session Weight - a great post from Bartłomiej Mucha confirms that Redis performance scales linearly with cache size, and since the session is stored in the cache then this is a problem. So scaling up Redis to the next tier will help a lot if you diagnose it as hitting the request per second limit for that cache size.

| improve this answer | |
  • do you have recommendations for how to tune or calculate the thread limits for an environment? – Mike J Apr 12 '19 at 20:16
0

I had this issue on 9.1 Update 1, none of the above worked, but when i added Hotfix 355701 and scaled up my App Service up from P1V2 to P3V2 and it went away.

| improve this answer | |
0

As @MarkGibbons mentions in his answer, session weight can be very heavy if proper maintenance and configuration is not in place. Recently, we faced a situation where the average session size was over 12MB and this was the leading cause for why we experienced these timeouts.

Review Session Data

Once connected to Redis, issue the commands:

  • slowlog get 10
  • Find a command with a large size, then issue hgetall {....._Data} (Key is pulled from slowlog output)

This will help identify why the session object is so large. It can be either private or shared session (it was both in our case).

Reduce Session Size

  • Remove old xDB data
  • Reduce MaxPageIndexThreshold setting on SetDummySession processor
    • This will reduce the private session size
  • Increase the Analytics.LongRunningRequest setting
    • When a long-running request occurs, it is logged to the session. Increasing the value reduces the likelihood of it being logged.
  • Change Analytics.LogLevel to None
    • Exceptions that occur during a user session are logged to session. Stack traces can eat up a considerable number of bytes. Changing to None will prevent them from being written to session.
  • Add a custom processor to <ensureSessionContext> pipeline
    • This is the same pipeline where the SetDummySession processor limits the session size. You can add a custom processor here to limit based on your needs. E.g. limit the number of page events per page to avoid spamming.

More details are available in my blog post.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.