Sitecore shutting down after Cache created: SqlDataProvider - Prefetch data(master)

Question

I am running Sitecore XM 10.0.1 in an AKS cluster. Everything has been running smooth for quite some time, but every once in a while the CM and CD containers just won't start.

The container logs just show a shutdown message, but always after the Cache created message for prefetch data from the master or web database (depending if it is CM or CD).

15060 08:47:50 INFO  Cache created: 'SqlDataProvider - Prefetch data(master)' (max size: 200MB, running total: 1179MB)
6504 08:48:40 INFO  *************************************************
6504 08:48:40 WARN  Sitecore shutting down
6504 08:48:40 WARN  Shutdown message: Initialization Error
HostingEnvironment initiated shutdown

I have set the startup probe to retry 30 times over 15 min, but it seems this shutting down message is showing every 2-4 minutes. The Pod itself is only restarted every 15 min because of that startup probe, so it is not Kubernetes which is restarting the application.

Has anyone seen this behavior before? Or is there any way I can figure out what is actually going on?

Setup

• Running on K8S version 1.20.5
• Sitecore XM 10.0.1
• Kubernetes hosted resources:
  1. CM
     • Requested MEM: 3Gi
     • Requested CPU: 1
     • Limit MEM: 3.5Gi
     • Limit CPU: 1
  2. CD
     • Requested MEM: 2Gi
     • Requested CPU: 1
     • Limit MEM: 3.5Gi
     • Limit CPU: 1
  3. ID
  4. Redis
  5. Solr
• AKS nodes: 2x Standard_DS4_v2 Windows nodes
• Databases hosted in a Azure ElasticPool
  • 100 DTU
  • Shared with a 2nd environment (both non-prod)
  • Max 10-15% DTU used, never reaching 100%
• Kubernetes doesn't seem to be reaching its limits
  • Overall CPU usage at 2%
  • Overall MEM usage at 21%
  • CD/CM Pod limits never going over 50% of CPU/MEM limits