Summary
Both Richard's and Pete's answers led me on a path to solve this. Pete's answer actually gave most of the solution, but I'll summarise my complete findings here.
On the use of Deployment Slots
Before I go on, I'll just touch on this point, as suggested by both. And while this can and will work around this problem, it doesn't actually solve the underlying problem - that you have an application that takes so long to fire up and eventually ends up on Azure's "bad list" for a while.
I see this solution in much the same way as I see caching as a solution for an under-performing component - it's a workaround for the issue.
That said, Deployment Slots are definitely valuable for a lot of reasons and I am still going to implement them on our environment. But I have effectively eliminated the original issue without them.
Roslyn Compilers
As suggested by Pete. The impact of this is massive, if executed correctly. And that does indeed mean that the instructions given on the two posts linked must be followed, including updating the system.codedom section of web.config.
In short you must:
- Install Roslyn (
Microsoft.Net.Compilers and Microsoft.CodeDom.Providers.DotNetCompilerPlatform). For 9.1.1 I ended up using version 2.9.0, as the latest version 3.1.0 required .NET framework 4.7.2 and we're on 4.7.1 (the latest version supported by Sitecore at the time of writing).
- I installed it as instructed, on all projects in our solution. I am not entirely sure this is necessary.
- Update your
web.config with the required system.codedom section. You will find it added to the web.config of all projects you added Roslyn to.
This, on it's own, took our startup/initialisation time to about 60% of what it was before, and more or less eliminated the 502 problem. But I pushed on.
Precompile Razor Views
I would suggest doing this. We only have one Razor view in our solution, so it made literally no difference for us (it's an SXA I-like-to-clone-Promo type solution). That said, if you have a more traditional Sitecore solution with many View files, this will make a difference.
This is already detailed here: Slow cshtml compilation on Azure WebApps
Deal with applicationInitialization
Thanks to @asmagin (Alex Smagin) on Slack, I learned about something I feel I should have been aware of, but was not. The applicationInitialization section you can add to web.config. This was the final piece of the puzzle, which effectively closed the book on the 502s. We've not had a single one since then.
Described in detail here: Application Initialization
In short, we added this configuration to system.webServer of web.config.
<applicationInitialization
doAppInitAfterRestart="true"
skipManagedModules="true"
remapManagedRequestsTo="applicationinit.htm">
<add initializationPage="/" hostName="#{contentDeliveryHost}#" />
</applicationInitialization>
Where applicationinit.htm was a short html file saying something like "Application is restarting, hang on". And #{contentDeliveryHost}# is just a token, being replaced with the full http address of the instance in question.
What happens here is, once the application starts initialising, the simple html page gets returned - which keeps the Azure load balancers happy so as to not hit the 230 second timeout. Even if the 230 second timeout was not really an issue after the steps above.
I'll tick Pete's answer for this question, as I was able to mine enough from it to actually solve the problem. But I'd like to point out that both Pete's and Richard's answers involve Deployment Slots and swapping of instances, warmups and so on - and none of this actually has any direct effect on solving this problem - that just works around it.
