1

I am trying to deploy the vanilla Sitecore packages to Kubernetes.

I have an issue mapping the domain into the cluster via the Ingress Controller. When I exec into the ingress controller pod, I can see the following log entries:

Error getting SSL certificate "sitecore/global-cd-tls": local SSL certificate sitecore/global-cd-tls was not found. Using default certificate
Error getting SSL certificate "sitecore/global-cm-tls": local SSL certificate sitecore/global-cm-tls was not found. Using default certificate
Error getting SSL certificate "sitecore/global-id-tls": local SSL certificate sitecore/global-id-tls was not found. Using default certificate

processed ingress via admission controller {testedIngressLength:1 testedIngressTime:0.051s renderingIngressLength:1 renderingIngressTime:0.001s admissionTime:26.2kBs testedConfigurationSize:0.052}
"successfully validated configuration, accepting" ingress="sitecore/sitecore-ingress"
"Found valid IngressClass" ingress="sitecore/sitecore-ingress" ingressclass="nginx"

"Adding secret to local store" name="sitecore/global-cd-tls"
"Adding secret to local store" name="sitecore/global-cm-tls"
"Adding secret to local store" name="sitecore/global-id-tls"

Yet the certificates in question have been generated and deployed as secrets in the same name space as the ingress resource and the controller.

I am using the following to deploy the controller:

.\helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx

.\helm repo update

.\helm install nginx-ingress ingress-nginx/ingress-nginx `
    --namespace sitecore `
    --set controller.replicaCount=2 `
    --set controller.nodeSelector."kubernetes\.io/os"=linux `
    --set defaultBackend.nodeSelector."kubernetes\.io/os"=linux `
    --set controller.admissionWebhooks.patch.nodeSelector."kubernetes\.io/os"=linux

Example ingress.yaml (cd and id removed for brevity):

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  namespace: sitecore
  name: sitecore-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/proxy-buffer-size: "32k"
    nginx.ingress.kubernetes.io/affinity: "cookie"
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-body-size: "512m"
spec:
  rules:
  - host: cm.globalhost
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: cm
            port: 
              number: 80
  tls:
  - secretName: global-cm-tls
    hosts:
    - cm.globalhost

I can confirm the following:

  • cm/cd/id pods are deployed and show TRUE for (Initialized/Ready/ContainersReady/PodScheduled)
  • kubectl get ingress returns (cd.globalhost, cm.globalhost, id.globalhost) with external IP from LB
  • kubectl get services returns a entry for (nginx-ingress-ingress-nginx-controller LoadBalancer (valid External IP))
  • I have ran the script to generate TLS .crt and .key files
  • I have deployed the secrets
  • I have created a local host entry mapping cm.globalhost to the external load balancer IP address for nginx
  • All other pods are healthy
  • Everything is deployed to same "sitecore" namespace

Navigating to site

When trying to navigate to the URL, Site can't be reached.

Within the ingress pod the following responds with 308 redirect:

curl -H "HOST: cm.globalhost" localhost

UPDATE 02/05/2023

Thanks for taking the time to look. Please see update as per your comments below:

All secrets are correctly named and match those in ingress yaml (see above): Secrets

Output from kubectl get secret global-cm-tls --namespace sitecore -o jsonpath='{.data}' shows base64 .key and.crt Key and Crt data

Logs from ingress pod below - note order of actions (1) create controller (2) deploy ingress (3) deploy secrets (I did it this way as described in sitecore docs - hence why secrets shown as missing initially).

-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v1.7.0
  Build:         72ff21ed9e26cb969052c753633049ba8a87ecf9
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.21.6

-------------------------------------------------------------------------------

W0502 12:39:18.009339       6 client_config.go:618] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0502 12:39:18.009515       6 main.go:209] "Creating API client" host="https://10.0.0.1:443"
I0502 12:39:18.043344       6 main.go:253] "Running in Kubernetes cluster" major="1" minor="24" git="v1.24.9" state="clean" commit="57fbbcc2804848b95cad5519f5ec9d6355430db9" platform="linux/amd64"
I0502 12:39:18.374623       6 main.go:104] "SSL fake certificate created" file="/etc/ingress-controller/ssl/default-fake-certificate.pem"
I0502 12:39:18.403976       6 ssl.go:533] "loading tls certificate" path="/usr/local/certificates/cert" key="/usr/local/certificates/key"
I0502 12:39:18.419703       6 nginx.go:261] "Starting NGINX Ingress controller"
I0502 12:39:18.426971       6 store.go:524] "ignoring ingressclass as the spec.controller is not the same of this ingress" ingressclass="nginx"
I0502 12:39:18.429045       6 event.go:285] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"sitecore", Name:"nginx-ingress-ingress-nginx-controller", UID:"4038b9c7-6cf6-41ae-b0fc-ce3b40bfc2ed", APIVersion:"v1", ResourceVersion:"4111862", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap sitecore/nginx-ingress-ingress-nginx-controller
I0502 12:39:19.621327       6 nginx.go:304] "Starting NGINX process"
I0502 12:39:19.621452       6 leaderelection.go:248] attempting to acquire leader lease sitecore/nginx-ingress-ingress-nginx-leader...
I0502 12:39:19.621959       6 nginx.go:324] "Starting validation webhook" address=":8443" certPath="/usr/local/certificates/cert" keyPath="/usr/local/certificates/key"
I0502 12:39:19.622139       6 controller.go:189] "Configuration changes detected, backend reload required"
I0502 12:39:19.626755       6 status.go:84] "New leader elected" identity="nginx-ingress-ingress-nginx-controller-95fc5f944-9jncx"
I0502 12:39:19.720627       6 controller.go:206] "Backend successfully reloaded"
I0502 12:39:19.720952       6 controller.go:217] "Initial sync, sleeping for 1 second"
I0502 12:39:19.721069       6 event.go:285] Event(v1.ObjectReference{Kind:"Pod", Namespace:"sitecore", Name:"nginx-ingress-ingress-nginx-controller-95fc5f944-pcshx", UID:"cab3c8f3-1267-441e-9b98-e3bd242b5e1a", APIVersion:"v1", ResourceVersion:"4115774", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
I0502 12:39:31.812432       6 store.go:433] "Found valid IngressClass" ingress="sitecore/sitecore-ingress" ingressclass="nginx"
W0502 12:39:31.812897       6 backend_ssl.go:47] Error obtaining X.509 certificate: no object matching key "sitecore/global-cm-tls" in local store
W0502 12:39:31.812928       6 backend_ssl.go:47] Error obtaining X.509 certificate: no object matching key "sitecore/global-cd-tls" in local store
W0502 12:39:31.812936       6 backend_ssl.go:47] Error obtaining X.509 certificate: no object matching key "sitecore/global-id-tls" in local store
I0502 12:39:31.812964       6 event.go:285] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"sitecore", Name:"sitecore-ingress", UID:"7698fe3d-7734-4bf3-82d5-c595f1420e10", APIVersion:"networking.k8s.io/v1", ResourceVersion:"4115869", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
W0502 12:39:34.257995       6 controller.go:1372] Error getting SSL certificate "sitecore/global-cm-tls": local SSL certificate sitecore/global-cm-tls was not found. Using default certificate
W0502 12:39:34.258040       6 controller.go:1372] Error getting SSL certificate "sitecore/global-cd-tls": local SSL certificate sitecore/global-cd-tls was not found. Using default certificate
W0502 12:39:34.258056       6 controller.go:1372] Error getting SSL certificate "sitecore/global-id-tls": local SSL certificate sitecore/global-id-tls was not found. Using default certificate
I0502 12:39:34.258148       6 controller.go:189] "Configuration changes detected, backend reload required"
I0502 12:39:34.362997       6 controller.go:206] "Backend successfully reloaded"
I0502 12:39:34.363893       6 event.go:285] Event(v1.ObjectReference{Kind:"Pod", Namespace:"sitecore", Name:"nginx-ingress-ingress-nginx-controller-95fc5f944-pcshx", UID:"cab3c8f3-1267-441e-9b98-e3bd242b5e1a", APIVersion:"v1", ResourceVersion:"4115774", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
W0502 12:39:37.591931       6 controller.go:1372] Error getting SSL certificate "sitecore/global-cm-tls": local SSL certificate sitecore/global-cm-tls was not found. Using default certificate
W0502 12:39:37.591981       6 controller.go:1372] Error getting SSL certificate "sitecore/global-cd-tls": local SSL certificate sitecore/global-cd-tls was not found. Using default certificate
W0502 12:39:37.591996       6 controller.go:1372] Error getting SSL certificate "sitecore/global-id-tls": local SSL certificate sitecore/global-id-tls was not found. Using default certificate
I0502 12:40:05.014665       6 store.go:588] "Secret was added and it is used in ingress annotations. Parsing" secret="sitecore/global-cd-tls"
W0502 12:40:05.014991       6 backend_ssl.go:47] Error obtaining X.509 certificate: no object matching key "sitecore/global-id-tls" in local store
W0502 12:40:05.015011       6 backend_ssl.go:47] Error obtaining X.509 certificate: no object matching key "sitecore/global-cm-tls" in local store
I0502 12:40:05.015565       6 backend_ssl.go:67] "Adding secret to local store" name="sitecore/global-cd-tls"
W0502 12:40:05.015709       6 controller.go:1372] Error getting SSL certificate "sitecore/global-cm-tls": local SSL certificate sitecore/global-cm-tls was not found. Using default certificate
W0502 12:40:05.015739       6 controller.go:1372] Error getting SSL certificate "sitecore/global-id-tls": local SSL certificate sitecore/global-id-tls was not found. Using default certificate
I0502 12:40:05.132452       6 store.go:588] "Secret was added and it is used in ingress annotations. Parsing" secret="sitecore/global-cm-tls"
I0502 12:40:05.133338       6 backend_ssl.go:67] "Adding secret to local store" name="sitecore/global-cm-tls"
W0502 12:40:05.133829       6 backend_ssl.go:47] Error obtaining X.509 certificate: no object matching key "sitecore/global-id-tls" in local store
I0502 12:40:05.255165       6 store.go:588] "Secret was added and it is used in ingress annotations. Parsing" secret="sitecore/global-id-tls"
I0502 12:40:05.257241       6 backend_ssl.go:67] "Adding secret to local store" name="sitecore/global-id-tls"
I0502 12:40:06.276504       6 event.go:285] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"sitecore", Name:"sitecore-ingress", UID:"7698fe3d-7734-4bf3-82d5-c595f1420e10", APIVersion:"networking.k8s.io/v1", ResourceVersion:"4116019", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I0502 12:40:11.796932       6 status.go:84] "New leader elected" identity="nginx-ingress-ingress-nginx-controller-95fc5f944-clqjt"

Ingress.yaml is set to use kubernetes.io/ingress.class: nginx (see question above).

My powershell script for deploying the ingress controller originally didnt have any reference to class (but I think nginx is default value). I updated the script and added the following lines further to your comments (but no success):

--set controller.ingressClass=nginx `
--set controller.scope.namespace=sitecore

Observations (based on the above)

I noticed in the log file shown the line:

"ignoring ingressclass as the spec.controller is not the same of this ingress" ingressclass="nginx"

Do you think this could be significant, given your comments about class? Also, I notice the lines like:

Adding secret to local store" name="sitecore/global-cm-tls

Then there are no further errors about the certificates. How would I confirm the certificates that are being used (i.e. is default still being used?)

Update

Please see below the response when I run the command from within the ingress pod:

openssl s_client -connect cm.globalhost:443 -servername cm.globalhost | openssl x509 -noout -text

Open Ssl command

Improve this question

1 Answer 1

Reset to default
2
+50

It looks like you have done most of the steps correctly. However, the error logs indicate that the Ingress Controller isn't able to find the SSL certificates even though they are present in the same namespace.

To help you further with this issue, I would suggest a few things to verify:

  1. Double-check the secret names and ensure they match with the names specified in the Ingress configuration: Run the following command to list all secrets in the sitecore namespace:
kubectl get secrets --namespace sitecore

Make sure the output contains the secrets global-cd-tls, global-cm-tls, and global-id-tls.

  1. Check the contents of the secrets to ensure they have the correct information:
kubectl get secret global-cm-tls --namespace sitecore -o jsonpath='{.data}'

This should display the base64 encoded data for the .crt and .key files. Make sure that both tls.crt and tls.key are present.

  1. Check the logs of the Ingress Controller to see if there are any issues with the secret handling:
kubectl logs -n sitecore -l app.kubernetes.io/name=ingress-nginx -c controller

Look for any logs related to the TLS secrets.

  1. Verify that you have the correct kubernetes.io/ingress.class annotation in your Ingress resource:

In your Ingress YAML file, you have the following annotation:

kubernetes.io/ingress.class: nginx
  1. Make sure that this annotation is consistent with the class specified during the installation of the Nginx Ingress Controller. The Ingress Controller needs to have the same class as the one defined in the Ingress resource.

Make sure the Ingress Controller is configured to use the correct namespace: By default, the Ingress Controller watches all namespaces. However, if you have specified a specific namespace for the Ingress Controller to watch during installation, it might not be able to access the secrets in the sitecore namespace.

Check the command used to deploy the Ingress Controller, specifically the --watch-namespace flag.

If you find any discrepancies or issues with the above suggestions, please fix them and try again. If you still face issues, you can provide additional logs and details to help diagnose the problem.

Updated 5/2/2023

To check the certificates being used by the ingress controller, you can access the website using a browser and inspect the certificate details. You should be able to confirm if the certificates being used are the ones you expect or if the default ones are still being used.

Another way to verify the certificates being used is to use the openssl command to check the certificate being served by the ingress controller. Replace with the actual domain name and run the following command:

openssl s_client -connect <your-domain>:443 -servername <your-domain> | openssl x509 -noout -text

Update 05/04/2023

Finally, we resolved issue by adding this setting into helm installation

--set controller.service.externalTrafficPolicy=Local

https://learn.microsoft.com/en-us/azure/aks/ingress-basic?tabs=azure-cli

enter image description here

The alternative way of fixing it - to use https://nip.io/ to avoid hosts file changes and this setting.

Improve this answer
8
  • Please see update above
    – Dean OBrien
    May 2 at 13:23
  • @DeanOBrien what was after these logs ?
    – Vadim Birkos
    May 2 at 15:09
  • I0502 12:40:06.276504 6 event.go:285] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"sitecore", Name:"sitecore-ingress", UID:"7698fe3d-7734-4bf3-82d5-c595f1420e10", APIVersion:"networking.k8s.io/v1", ResourceVersion:"4116019", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync I0502 12:40:11.796932 6 status.go:84] "New leader elected" identity="nginx-ingress-ingress-nginx-controller-95fc5f944-clqjt"
    – Vadim Birkos
    May 2 at 15:09
  • updated an answer @DeanOBrien
    – Vadim Birkos
    May 2 at 15:12
  • 1
    Thanks for all your help with this. The setting externalTrafficPolicy=Local seems to be a new addition, as certainly wasnt required last year. Also seems a bit confusing in my opinion.. that note suggests by enabling it TLS pass through wont work? Also, docs suggest just a lack of IP being available - not that it would block all ingress.
    – Dean OBrien
    May 4 at 8:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.