Azure PaaS reports certificate not found

Question

I have deployed Sitecore 9.0.2 XP Single in an Azure App Service Environment (ASE). After deployment I keep seeing the following error being reported in the log files:

Exception: System.InvalidOperationException
Message: **The certificate was not found**.
Source: Sitecore.Xdb.Common.Web
   at Sitecore.Xdb.Common.Web.CertificateWebRequestHandlerModifier.Process(HttpClientHandler handler)
   at Sitecore.Xdb.Common.Web.CommonWebApiClient`1.CreateRequestHandler()
   at Sitecore.Xdb.Common.Web.CommonWebApiClient`1.CreateClient()
   at System.Lazy`1.CreateValue()
   at System.Lazy`1.LazyInitValue()
   at Sitecore.Xdb.Common.Web.CommonWebApiClient`1.d__37.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Sitecore.Xdb.Common.Web.CommonWebApiClient`1.d__32.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Sitecore.XConnect.Client.WebApi.ConfigurationWebApiClient.d__4.MoveNext()

I used these Quickstart templates that Sitecore provides on Github, and only modified them slightly to allow the use of the ASE.

I generated a self-signed client certificate. In the ARM template parameter file I used the 'allowInvalidClientCertificates' parameter and set it to 'true'. I see that the certifcate has been processed during the installation. The Resource Group contains an object of type Microsoft.Web/certificates. I also see that the Web.config file contains the following line:

<add key="AllowInvalidClientCertificates" value="True" />

The ConnectionStrings.config contains entries listing the certificate thumbprint. As suggested in this stackexchange thread, I added "AllowInvalidClientCertificates=true" to the connection string. But that didn't solve the problem.

<add name="xconnect.collection.certificate" connectionString="StoreName=My;StoreLocation=CurrentUser;FindType=FindByThumbprint;FindValue=DUMMYTHUMBPRINT;AllowInvalidClientCertificates=true" />
<add name="xdb.marketingautomation.operations.client.certificate" connectionString="StoreName=My;StoreLocation=CurrentUser;FindType=FindByThumbprint;FindValue=DUMMYTHUMBPRINT;AllowInvalidClientCertificates=true" />
<add name="xdb.marketingautomation.reporting.client.certificate" connectionString="StoreName=My;StoreLocation=CurrentUser;FindType=FindByThumbprint;FindValue=DUMMYTHUMBPRINT;AllowInvalidClientCertificates=true" />
<add name="xdb.referencedata.client.certificate" connectionString="StoreName=My;StoreLocation=CurrentUser;FindType=FindByThumbprint;FindValue=DUMMYTHUMBPRINT;AllowInvalidClientCertificates=true" />

Any suggestion would be highly appreciated that could lead to a solution. Many thanks in advance.

Answer 1

Answering my own question here. Not for points but in case someone else runs into the same problem.

It turns out the error message "The certificate was not found" doesn't refer to the Sitecore client certificate, but refers to the SSL server certificate that was configured on the ASE. Let me explain further.

We deployed an Internal ASE (ILB) and configured it with a non-public Top Level Domain. We also configured a self-signed certificate on the ASE, because you can only request valid 3rd party certificates for public TLD's. Using an SSL certificate from an internal CA wasn't possible, because you cannot simply import the root cert from the CA in a Web App's certificate store, like you do on a Windows server. At least, not that I am aware of.

The SSL certificate you configure on an ASE gets configured on the load balancer of that ASE. All traffic to Web Apps flows through the ASE load balancer. This also applies for Web Apps that communicate which each other that live inside the same ASE. In our situation, Sitecore CD wants to communicate to the xConnect Web App, then hits the load balancer that presents the self-signed SSL certificate. And apparently, Sitecore doesn't like self-signed SSL certificates. And there's no way I found to let Sitecore ignore invalid or invalid certificates, like with a web browser.

The solution was easy. We configured the Web Apps with hostnames (URL's) using a valid TLD and configured a 3rd party issued SSL certificate. As a last step we changed configuration of the ConnectionStrings.config to reflect the new hostnames. That's it. The errors disapeared from the logging and we are able to successfully use xConnect.

Answer 2

In your Azure Portal, select your web app running the site.

Go to the SSL Settings blade, and in the Private Certificates tab check whether you've got the xConnect client certificate installed there, and whether that thumbprint matches what you have in your config settings.
(If the certificate doesn't show here it needs to be uploaded first)

If that's the case, open the Advanced Tools (should have a link that opens in a new tab), where you can select Debug console -> PowerShell. In the PowerShell console, type

Get-ChildItem -path cert:\CurrentUser\My

See if your certificate appears there.

If that's not the case, go back to your web app, go to Application Settings and add the setting WEBSITE_LOAD_CERTIFICATES with the value of either the thumbprint or (if you have more certificates) the value *: