6

I've noticed the 502 error after deployment to Azure question, but it appears to be related to a failed deployment and rollback of databases - not what we're seeing here.

I am getting the above

502 - Web server received an invalid response while acting as a gateway or proxy server

It happens after any completion of our Azure DevOps pipeline that deploys to the WebApp. Here's what I've observed:

  • It happens on serving any of the "public" sites on the solution. So if I call up a page from an SXA site or a regular site, I get about a 5 minute delay while it's loading and then ultimately giving up.
  • It does not happen when I reach the "internal" sites on the solution. Content Editor for instance. It does happen if I try and start the Experience Editor.
  • Strangely, "Smiling Asian Lady" (default Website) loads up almost instantly. The problem may be isolated to SXA

After the 5-10 minute wait and the initial 502 error being shown, I can refresh the page and Sitecore proceeds to serve up the requested page as if it went through a normal cold-start. So roughly 30 seconds.

edited to clarify/rectify

We don't have an explicit AppGateway in front of our sites. We do however use Auto Scaling (not dynamic, but set to 2 instances) which would mean that the scaling has an internal load balancer in place. Which I guess acts in much the same way as a gateway would do.

Can anyone point me to what's going on here?

7

Background Information

You ask if anyone can point you to what's going on. This is a COMMON symptom for all Sitecore sites running in Azure PaaS Web App Services.

The issue stems from the fact that the underlying load balancer that sits between the App Service and the public internet has an internal probe that checks to see if the application is running and serving. When this times out after 230 seconds, it shows that error. Sitecore takes quite a bit of time on startup before it accepts a web request. Therefore, the load balancer check is faster than Sitecore startup and reports the app service as down, and halts requests going to it.

Options Available

Runtime Compilation of Views

This is the number one culprit of slow Sitecore startup.

  1. Use the latest Roslyn compiler (make sure you have the system.codedom node in the right place in your web.config).
  2. Precompile your Razor views

Using Deployment Slots

This option, as mentioned in another answer, allows you to deploy code in a staging environment, do the web application start, allow Sitecore to start up, and then essentially swap the dns entries for the site to the active slot. To be very clear on this point, Deployment Slots are individual instances of sitecore running. Both slots run at the same time. So if you have a release of new items and changes that are dependant upon code being in place, then you have to have separate databases for each slot, which turns into a management issue. This can also mimic a blue/green deployment strategy. But as mentioned before, you have to take the databases (and ultimately what to do with them) into consideration.

A great article on exactly how staging slots warm-up works is here.

Use an Azure Application Gateway

These provide configuration settings to allow you to use a custom error page instead of displaying the default on that Azure PaaS uses. You can also enable WAF on the Azure Application Gateway (supported in Sitecore 9.1+) for some extra security, although it's not cheap.

Scaling Up your Web App Service

My experience thus far has been that Sitecore's guidance for Azure PaaS sizing is significantly low. Pumping up the scale of your Web App Service has dramatic effects to the speed of Sitecore, especially on application start. It can also have dramatic effects on your operational budgets, however for most people the main point to note is that an S2 is equal to a P1V2 in cost, but P1V2 is far superior to S2 as it uses SSD's and has better compute power, not to mention the additional features you get from Premium.

  • Does it mean that it takes more 230 seconds for Sitecore App Service to serve the first request after startup? I can't believe that it's taking so long. – grg Jun 19 at 11:09
  • 1
    When uaing the standard scaling Sitecore recommends, its taking nearly 5 minutes. – Pete Navarra Jun 19 at 16:11
  • 1
    @grg I had a site go from 8 minutes startup to 50 seconds after installing the Roslyn compiler. I assume that the default one has some major issues on azure, which are fixed in the Roslyn one. – Mark Gibbons Jun 19 at 21:53
5

From what you have said, I think you need to look into using deployment slots on the app service.

When you deploy to the app service, it is likely that you are getting an application recycle and while the application warms up, the gateway will throw 502 errors for a while until the application is completely up and running.

So solve that, add a deployment slot to the application service and make sure your release pipeline is deploying to the non-live slot. Azure will give you a url to hit that slot and warm it up. Then you can run any tests etc... you need to on the stage slot. Finally, once you are happy you can swap the slots and promote the stage slot to production. This will prevent any downtime on the production slot.

enter image description here

Release Pipelines

Depending on what you are using to release, you will need to add steps to your release pipeline to deploy to your staging slot, warm up the staging slot application and finally swap the slots. Azure DevOps has good tooling for this.

Considerations

With slots you are partially on your way to green/blue deployments. So you should decide whether you are going to have green/blue web databases and/or indexes. If you will need to add steps in to swap the connection strings on you CM and publish/re-index etc...

5

Summary

Both Richard's and Pete's answers led me on a path to solve this. Pete's answer actually gave most of the solution, but I'll summarise my complete findings here.

On the use of Deployment Slots

Before I go on, I'll just touch on this point, as suggested by both. And while this can and will work around this problem, it doesn't actually solve the underlying problem - that you have an application that takes so long to fire up and eventually ends up on Azure's "bad list" for a while.

I see this solution in much the same way as I see caching as a solution for an under-performing component - it's a workaround for the issue.

That said, Deployment Slots are definitely valuable for a lot of reasons and I am still going to implement them on our environment. But I have effectively eliminated the original issue without them.

Roslyn Compilers

As suggested by Pete. The impact of this is massive, if executed correctly. And that does indeed mean that the instructions given on the two posts linked must be followed, including updating the system.codedom section of web.config.

In short you must:

  • Install Roslyn (Microsoft.Net.Compilers and Microsoft.CodeDom.Providers.DotNetCompilerPlatform). For 9.1.1 I ended up using version 2.9.0, as the latest version 3.1.0 required .NET framework 4.7.2 and we're on 4.7.1 (the latest version supported by Sitecore at the time of writing).
  • I installed it as instructed, on all projects in our solution. I am not entirely sure this is necessary.
  • Update your web.config with the required system.codedom section. You will find it added to the web.config of all projects you added Roslyn to.

This, on it's own, took our startup/initialisation time to about 60% of what it was before, and more or less eliminated the 502 problem. But I pushed on.

Precompile Razor Views

I would suggest doing this. We only have one Razor view in our solution, so it made literally no difference for us (it's an SXA I-like-to-clone-Promo type solution). That said, if you have a more traditional Sitecore solution with many View files, this will make a difference.

This is already detailed here: Slow cshtml compilation on Azure WebApps

Deal with applicationInitialization

Thanks to @asmagin (Alex Smagin) on Slack, I learned about something I feel I should have been aware of, but was not. The applicationInitialization section you can add to web.config. This was the final piece of the puzzle, which effectively closed the book on the 502s. We've not had a single one since then.

Described in detail here: Application Initialization

In short, we added this configuration to system.webServer of web.config.

    <applicationInitialization
        doAppInitAfterRestart="true"
        skipManagedModules="true"
        remapManagedRequestsTo="applicationinit.htm">
        <add initializationPage="/" hostName="#{contentDeliveryHost}#" />
    </applicationInitialization>

Where applicationinit.htm was a short html file saying something like "Application is restarting, hang on". And #{contentDeliveryHost}# is just a token, being replaced with the full http address of the instance in question.

What happens here is, once the application starts initialising, the simple html page gets returned - which keeps the Azure load balancers happy so as to not hit the 230 second timeout. Even if the 230 second timeout was not really an issue after the steps above.

I'll tick Pete's answer for this question, as I was able to mine enough from it to actually solve the problem. But I'd like to point out that both Pete's and Richard's answers involve Deployment Slots and swapping of instances, warmups and so on - and none of this actually has any direct effect on solving this problem - that just works around it.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.