After applying the Q3 2016 Security Update our sites went down

Question

As you all might know Sitecore issued a critical security hotfix, after applying the fix to our client sites, most of them went down for on average 30 minutes before going back up again, and one of them is still down, does anyone know what is causing this?

I checked the Sitecore Logs for one of our sites, and the only exception that caught my eye was the following, but I still doubt that it is the cause:

ERROR SessionEndPipeline failed.
Exception: System.ArgumentNullException
Message: Value cannot be null.
Parameter name: owner
Source: Sitecore.Kernel
   at Sitecore.Diagnostics.Assert.ArgumentNotNull(Object argument, String     argumentName)
   at Sitecore.Diagnostics.Log.Warn(String message, Object owner)
   at Sitecore.WFFM.Core.Extensions.Warn.IsNull(Object obj, String name)
   at Sitecore.WFFM.Analytics.AnalyticsTracker.RegisterFormDropouts()
   at (Object , Object[] )
   at Sitecore.Pipelines.PipelineMethod.Invoke(Object[] parameters)
   at Sitecore.Pipelines.CorePipeline.Run(PipelineArgs args)
   at Sitecore.Pipelines.CorePipeline.Run(String pipelineName, PipelineArgs     args, String pipelineDomain, Boolean failIfNotExists)
   at Sitecore.Pipelines.CorePipeline.Run(String pipelineName, PipelineArgs     args, String pipelineDomain)
   at Sitecore.Pipelines.CorePipeline.Run(String pipelineName, PipelineArgs     args)
   at Sitecore.Pipelines.EndSession.SessionEndPipeline.Run(SessionEndArgs     sessionEndArgs)
   at Sitecore.Web.Application.RaiseSessionEndEvent(HttpApplication context)

Answer 1

As the comments on the question rightly point out; there could be a number of potential sources for this event that happened to you. As such, I don't know if a real answer could ever be found.

That said, I will attempt an answer. What you're getting here is just my experience summed up to the best guess I can come up with. I will also provide you means to verify.

Here's the things you said that I will factor in:

after applying the fix to our client sites

How many? To me, this particular wording suggests to me that you have a medium-to-large client count and potentially all these sites are hosted in an environment you control.

most of them went down for on average 30 minutes before going back up again

This tells me it's variable; and site dependent. And in so, we can pretty much already rule out the patch itself - if it was a direct cause of the delay, it would be relatively equal for all sites.

Applying the patch, however, restarts the application pool

I checked the Sitecore Logs for one of our sites, and the only exception that caught my eye was the following

This exception does not seem related to what you're experiencing. And if there are no other log entries as you state, we can conclude the sites weren't mass failing while this event took place.

So with all of these assumptions in place; I think - individually - your client sites are "slow starters". While there could be any number of reasons for this; more often than not it comes down to sub-optimal information architecture and poorly designed components that query too heavily. Under these conditions, a "1st pagerequest" is usually dis-proportionally slow, subsequent requests are "fine" since Sitecore has now managed to cache all of this.

To worsen the problem; you may have applied the patch rapidly. To all of the sites. All at once.

And if my line of reasoning above holds true; this now means X number of sites resetting their application pools, performing sub-optimal queries en masse - completely flatlining your underlying SQL servers, not to mention CPU and memory resources of your web servers.

And under these circumstances; a 30 minute or more recovery time from an application pool reset is entirely likely.

To validate if my theory holds true; try resetting the apppool on 1 of your sites. Then time how long it takes for the site to respond. Then reset 2 apppools and time the first site again (after requesting home page on both). If it takes longer than the first time, this would be the smoking gun as they say.

And if not; this is the information I believe we'll need - to be able to provide a better answer.

• Your hosting/infrastructure setup. VMs? Shared SQL servers?
• How many sites were updated and how quickly?
• "Cold Start" request time for your home page, after an app pool reset
• Number of items in your [EventQueue] and [History] tables

This is the best I can come up with, on the given information. I hope it helps you.