Content Delivery Role on Sitecore 9 throws Post condition failed

Question

We deployed our codebase on the upgraded instance on Sitecore 9.0.2 in a scaled environment with 1 CM (for CM, Processing, Reporting), 1 xConnect and 2 CD (for Content Delivery Role). The CM and all the xConnect Roles are coming up fine but the Content Delivery Instance is throwing the exception mentioned below.

Using octopus to transform the config files, we verified that role:define has the correct values on the corresponding environments. On Sitecore 9 there is no SwitchMasterToWeb config file, hence the enabling/disabling references to Master DB should not be a manual process if we have role:define as ContentDelivery.

We also tried manually removing the Master DB configuration from Sitecore.Config, but that didn't help as well.

We referred on this question as well but no luck: Post Condition Failed on Sitecore 9 Content Delivery Server

What else are we missing here?

Exception: System.InvalidOperationException
Message: Post condition failed
Source: Sitecore.Kernel
   at Sitecore.Diagnostics.Assert.ResultNotNull[T](T result, String message)
   at Sitecore.Configuration.DefaultFactory.GetDatabase(String name)
   at Sitecore.Configuration.DefaultFactory.GetDatabases()
   at Sitecore.Data.Managers.LanguageProvider.InitializeEventHandlers()
   at Sitecore.Data.Managers.LanguageProvider..ctor(BaseCacheManager cacheManager, BaseSettings settings)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ConstructorCallSite.Invoke(ServiceProvider provider)
   at Microsoft.Extensions.DependencyInjection.ServiceProvider.ScopedCallSite.Invoke(ServiceProvider provider)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ConstructorCallSite.Invoke(ServiceProvider provider)
   at Microsoft.Extensions.DependencyInjection.ServiceProvider.ScopedCallSite.Invoke(ServiceProvider provider)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider provider, Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider)
   at Sitecore.DependencyInjection.LazyResetable`1.get_Value()
   at Sitecore.Globalization.Language.TryParse(String name, Language& result)
   at Sitecore.Globalization.Language.Parse(String name)
   at Sitecore.Jobs.JobOptions..ctor(String jobName, String category, String siteName, Object obj, String methodName, Object[] parameters)
   at Sitecore.Jobs.JobOptions..ctor(String jobName, String category, String siteName, Object obj, String methodName)
   at Sitecore.CES.DeviceDetection.Providers.DeviceDetectionClient.Tick()


ManagedPoolThread #3 11:32:53 INFO  Trying to load XML configuration /App_Config/Security/Domains.config
ManagedPoolThread #0 11:32:53 ERROR Exception in alarm clock event subscriber.
Exception: System.InvalidOperationException
Message: Post condition failed
Source: Sitecore.Kernel
   at Sitecore.Diagnostics.Assert.ResultNotNull[T](T result, String message)
   at Sitecore.Configuration.DefaultFactory.GetDatabase(String name)
   at Sitecore.Configuration.DefaultFactory.GetDatabases()
   at Sitecore.Eventing.EventProvider.RaiseQueuedEvents()
   at Sitecore.Data.Eventing.EventQueueMonitor.AlarmClock_Ring(Object sender, EventArgs e)
   at Sitecore.Services.AlarmClock.<>c__DisplayClass11_0.<Heartbeat_Beat>b__0(Object state)

Answer 1

As suggested by Richard and Pete in comments above the issue was related to errors in configuration files post variable transformation. We analyzed the following to figure out the root cause:

1. Since it is an upgraded instance we reviewed the role:define and role:require in the solution to make sure nothing is misconfigured for CD Role.
2. Revisited the octopus variable transformations.
3. Reviewed the output on showconfig.aspx on CD Roles.

In our case it ended up with a configuration conflict as a result of octopus variable transformation. Though not recommended but we have couple of config edits within OOTB Sitecore configuration files rather having patch files. We found that the transformations taking place for Sitecore.Buckets.config was the root cause, as it was throwing errors for Master Database and Pre-Prod Database on the Conetnt Delivery Role. Hence we removed the octopus transformation for the Sitecore.Buckets.config file and added a patch file.