We are suddenly experiencing a problem with Sitecore never starting up. It looks like the last meaningful line in the log is that the IndexingStateSwitcher job starts, however, it never ends. In our other environments this job takes ~1 second to run. See log line below:
INFO Job started: Sitecore.ContentSearch.SolrProvider.Agents.IndexingStateSwitcher
After approximately 40 mins, we see an entry in the system Event Viewer that the application pool failed to respond to ping, at which point the application pool is stopped and a new process is started. Here is what appears in the event viewer:
A process serving application pool 'prod.XXXX.org' failed to respond to a ping. The process id was '4328'.
This may occur, 3-5, even 6 times and eventually the application starts up and behaves as expected. We confirmed that connections to SOLR work without issue once Sitecore starts up. SOLR has plenty of memory and responds to queries without issue. The issue does not seem to coincide with a code or configuration deployment in Sitecore.
Are there any reasons this job may hang like that?
In cases when Sitecore starts up without issue, after a few retries, the job starts and finishes in just 1 second like on any other environment. The issue appears to be intermittent.