Most important caveats for continuous delivery and zero downtime

Question

Currently I am working on the physical view of our Sitecore architecture, part of this view is how we do deployments which cause no downtime for endusers and preferably our content editors. So far, we have automated all of our deployments, because we have 20 different sitecore instances (all existing of 4x CD and multiple CM's) and we have tons of deployments a day.

From the CI and automated deployments, our next step is to be able to deliver zero downtime deployments. Multiple strategies have come into my mind and most strategies fit a specific situation, which will work or won't work for different situations, so I am not looking for the one-size-fits-all solution, but for the challenges that we will face.

From what I've come up with, I can think of the following:

• User Sessions. Graceful remove CD's from a load balancer or don't use sticky sessions
• databases. what data is stored in web/core when someone visits a CD server. Is there any data that is stored? (I know that, for example, userprofiles, passwords, groupmemberships et cetera) are stored in the core database. Are there any queues that I have to take care of?
• WFFM/other modules -> where do they store data? I believe they use some queues as well?
• Itemdata: develop defensively, make sure that renderings (and other logic) don't freak out when a field is removed/renamed.

Answer 1

Sitecore is a great CMS to use for zero downtime, here are some thoughts:

Have multiple publishing targets

This will solve the issue of fields being renamed etc. Details here https://sitecore.stackexchange.com/a/1413/87

User sessions

Ideally you're going to want to shove the session into the database, perhaps Mongo for the latest versions 8.x onwards or SQL is fine too, then you don't need to worry about sessions getting blown away.

However there are some sites where the session has been abused to store all kinds of wonders that stops it serialising to disk. In this scenario you can do two things.

• Drain boxes from the load balancer before deployments
• Use app_offline.htm to get rid of users gracefully https://stackoverflow.com/a/1153545/861315 but you will need a way of ensuring your load balancer drops the box out to stop users getting the app_offline.htm

WFFM

Most modules will use their own databases, so this won't cause you problems

CM box

This is the only sticking point really, unless you have multiple CM servers then there is always going to be a blip here, unless you deploy side by side in IIS and just flip over to serve the new code - for this to work you need to be confident that you can trust your release to be dropping all code each release (or virtualise the Sitecore folders and just drop production code each time)

Core and Web databases

These are pretty robust and not something you'll need to worry about - or at least nothing I can think of.

Answer 2

For our zero-downtime setup we clone the CD servers, the Web database and the SOLR cores at every production release. In a cloud setup like Azure this is pretty easy and cheap. On prem not so much. For efficiency, we don't clone all the SOLR cores, only the ones containing content in the main Sitecore content tree:

• sitecore_web_index_rYYMMDD
• sitecore_marketing_asset_index_web_rYYMMDD
• sitecore_fxm_web_index_rYYMMDD
• social_messages_web_rYYMMDD

The idea is that you get a complete blue/green scenario, and you can stand up new functionality that potentially modifies shared templates or components in the DB or indexes without worrying about downstream dependencies in the code, like Glass classes.

We have automated the application of configuration changes and new hardware creation to make this process as friction-free as possible, but we still need a brief outage of the Auth server whose app pool must be recycled to pickup the new configuration settings, and we need to block publishing to production for about 2-3 hrs while we clone the DBs and indexes (we have very big DBs).

Once the new equipment is provisioned we perform some smoke testing and can swing the traffic across by modifying the load balancer settings. Then we remove the old servers and power them down.

Should a critical issue be discovered we can easily roll back to a last-known-good state by simply powering up the old servers and swinging the traffic back through the load balancer, which takes about 20 minutes tops.