Outage on your sitecore website during an azure platform upgrade

Question

Our website becomes unresponsive and we get Pingdom alerts whenever there is either an azure platform or infrastructure upgrade on our website. This happens once a month.

We have an app gateway and our Sitecore instance (with 3 instances) is behind the App gateway in Azure

We have contacted Azure multiple times but was of no much use. We contacted Sitecore and they provided with the following patch - https://kb.sitecore.net/articles/910936 but even that did not work.

As anybody faced a similar issue. if so how did you resolve the same?

Note :

1. Pingdom would ping our website from different regions every 1 minute and would consider down after 30secs of timeout

2. We have analyzed app insights and we don't find anything suspicious that can cause such outages only thing that we see that whenever we have any kind of upgrade activity out Sitecore website goes down for around 30 seconds and Pingdom alerts the same.

3. Our website is hosted in only one region in azure.

Below are some of the screenshots explaining about the outages that I mean above:

Do let me know if you need any other details?

Thanks,

Answer 1

We also faced the same issues but unfortunately isn't much we can do. Here is some further explanation from what I heard from Azure Support yesterday.

There are two types of updates that occur

1. App service specific updates

These updates are to allow for bug fixes, security patching, enabling new functionality, and they are deployed regionally in batches, and in a round robin way.

The way to be "protect" against these updates is either

• To have your App Service Scaled Out, because it will restart one instance at a time which gives the instance enough time to be back up once the updates starts in the another one
• An App Service in a different region (or different scale unit by using a different App Service Plan) and rely on Azure Traffic Manager (as stated in the image you provided)

2. Storage level updates

These updates are done on the file servers themselves. They also allow for improved durability and security.

Now, with storage updates there are different things that happen

Any App Service that is sitting on Storage that is updated will be restarted twice, and this is by design.

And you also may noticed, at least happened with our App Services, there was an Environment Variables Change, and this is due to the file servers being updated, and this is what happens in Azure Infrastructure (completely behind the scene)

The files are stored on remote storage, and each App Service has two storage accounts that are tied to it: a primary (active), and a secondary (passive). When the storage that is hosting the primary is updated, the App Service can't acces it, so Azure has to point the App Service at the secondary location of files hence the App Service is restarted. And it restarts again, when the update is complete, and the storage is changed it back as it was.

And for Storage updates, it does not matter if you have a Scale Out App Service or not, all instances will be restarted at the same time

The way to be "protected" against these storage update is

• To have an App Service in a different region (or an additional App Service but using different App Service Plan), and rely on Azure Traffic Manager to switch the traffic
• And as Azure states, the use of Local Cache but you might find issues to use with Sitecore: max 2GB of local cache, and Sitecore doesn't support it

Answer 2

we faced the same issues. During an azure platform upgrade, a new app service instance is created, added to the pool, while the old instances are removed. THese instances are not warm, so there is a period where traffic will land, but will not get response, as Sitecore is still initializing.

an option is to add application initialization to your web.config, as described over here. Please note that this approach might not completely work for you, as you need to tweak the initialization. THe init happens from the instance itself, so it has to be able to access "localhost", might (or might not) be https only, et cetera.