3

We have our SOLR server running on an Azure App Service. Occasionally the cores will lock. This can be resolved by a restart but I am hoping that there is some configuration that can help to prevent this. Did anyone experience this?

The SOLR is running as a single instance with no scaling or clustering.

sitecore_core_index: org.apache.solr.common.SolrException:org.apache.solr.common.SolrException: Index dir 'D:\home\site\wwwroot\server\solr\sitecore_core_index\data\index/' of core 'sitecore_core_index' is already locked. The most likely cause is another Solr server (or another solr core in this server) also configured to use this directory; other possible causes may be specific to lockType: native
4
  • 1
    Using solr in azure web apps is not recommended. Dec 16, 2019 at 8:15
  • I get that and we do have a plan to SearchStax in the future. Was hoping there might be an interim solution to make it more stable
    – JKerny
    Dec 16, 2019 at 22:53
  • 1
    Unfortunately Azure Web Apps is not the right choice for Solr to begin with. You are much better off spinning up a dummy VM for now. A Dev instance of SearchStax Solr is actually priced very well, to a point that it's not worth the headache going through any other processes. Dec 17, 2019 at 4:38
  • I have the same issue with Sitecore 9.0.2 and solr, indexes are just spontaneously locking and I have to restart solr server to get everything up and running again. While crawling the logs I've found out that XConnect is probably the reason. Perhaps, it's making too many requests per seconds. This is just my assumption. Have you found the solution to solve it? Mar 12, 2020 at 15:49

5 Answers 5

4

I've just blogged about this after talking to Sitecore support regarding this issue, they said:

  1. Make sure the Azure Web App is not scaled out (instance count should be set to 1).
  2. Make sure the environment variable WEBSITE_DISABLE_OVERLAPPED_RECYCLING is set to 1

We already have our App Service scaled to just one instance but when checking the configuration settings we didn’t have the WEBSITE_DISABLE_OVERLAPPED_RECYCLING setting set. So we added this config and set the value to 1.

enter image description here

This resolved most issues with Solr locking, more info here:

http://www.flux-digital.com/blog/fixing-solr-azure-app-service-core-locking-issues/

Update: I still had issues with cores locking occasionally and started on a solution for it using Powershell in a Runbook a while ago and I have now completed this so have blogged here about it: https://www.flux-digital.com/blog/auto-re-starting-solr-indexes-get-locked-azure-app-service/

The above solution is based on @BikerP's comment below.

2

You can spin up a Solr using Azure Container Instances for much less than the price of a VM. It is a bit slow, but fine for non-production systems, and you can have a new one in about 5 minutes flat.

This is backed onto an Azure Storage File Share where the actual cores can live.

The PowerShell script below will set you up for less than $50 a month (remember to turn it off when you're not using it). We have other scripts to start and stop the containers every weekday, which saves about 70% of the price.

You should check on https://hub.docker.com/_/solr for the version closest to what your Sitecore will need. Definitely stay on the same major version, and try to stay on the same minor version. Revision is not so important.

$resGrp   = "development"
$stgAcc   = "mydevstorage"
$stgKey   = "***************************************=="
$stgShareName = "proj123"
$location = "australiaeast"
$image    = "solr:8.1-slim"
$cpus     = 1
$maxRAM   = 2
$solrMem  = ( $maxRAM * 1024 - 256 )

New-AzContainerGroup -ResourceGroupName $resGrp -Name "my-dev-solr" `
   -Image $image -Cpu $cpus -MemoryInGB $maxRAM `
   -DnsNameLabel "my-dev-solr-123" -Port 8983 `
   -Location $location -OsType Linux -RestartPolicy OnFailure `
   -EnvironmentVariable @{ "SOLR_HOME" = "/data/solr"; "SOLR_JAVA_MEM" = "-Xms256m -Xmx$($solrMem)m" } `
   -AzureFileVolumeShareName $stgShareName -AzureFileVolumeAccountCredential ( New-Object PSCredential -ArgumentList @( $stgAcc, (ConvertTo-SecureString $stgKey -AsPlainText -Force )) ) `
   -AzureFileVolumeMountPath "/data"

But what does this all do?

Glad you asked!

Mostly it's self explanatory but you need an Azure storage account with a "File Share" (not Blob Storage) called, in this case proj123. In that share create a folder solr and put your core folders in that. In the root solr folder you also need the appropriate solr.xml file. You can copy all that stuff from your web app.

One thing though - this is a public-facing, unencrypted, unsecured Solr, so you need to add some Basic authentication as well.

Create a file called security.json and put it in the root with solr.xml. The contents needs to look like this:

{
 "authentication":{
   "blockUnknown":true,
   "class":"solr.BasicAuthPlugin",
   "credentials":{"solr":"***************"},
   "forwardCredentials":false,
   "":{"v":0}
  },
  "authorization":{
    "class":"solr.RuleBasedAuthorizationPlugin",
    "user-role":{"solr":"admin"},
    "permissions":[{"name":"security-edit","role":"admin"}]
  }
}

More detail about the contents of that file (and a default password you can use) including how to change the password using a URL, can be found in the Solr Docs.

Then in Sitecore you need to update your URLs to be http://user@pwd:hostname:8983/solr but there's also more info about that in the Sitecore docs.

1

We can configure the health check in the Azure web app to ping if you are using multiple instances the index by following the steps below

  1. Go to your Solr web app in Azure Portal

  2. Under Monitoring, select Health Check

  3. Enable Health Check

  4. Set health Probe path to

    /solr/sitecore_master_index/admin/ping

  5. Set Unhealthy instance removal --> Load balancing threshold to 2 mins (thats the minimum)

enter image description here

  1. Save the changes.

If the ping response is not 200, then Azure will restart the app.

Note: Saving this setting will restart the Solr web app

If you are using a Single instance, you can set up Auto-heal as a workaround

more details here https://sitecorepro.com/fix-solr-core-locking/

Hope this helps!

1
  • This will not work as expected when you "basic authentication" is configured in Solr. You can do two things, either configure Solr to allow anonymous access to the core ping url, or just use "/solr/img/solr.svg" for the Health probe path.
    – Saab
    Apr 15 at 7:41
1

This is a known issue with Solr on Azure PaaS, my imperfect solution has been to write a PowerShell script that runs on an interval to check the returned JSON from the Solr core status endpoint - to see if it contains the string SolrException, if present the script restarts the app service. I run the script every 10 minutes and I find it does a restart on average about every 2 weeks or so, although this can vary a lot. As I said, it's not perfect, but it works for me.

https://xxx.azurewebsites.net/solr/admin/cores?action=STATUS

You could also do the same with a ping on a specific index too, looking for string Status:OK:

https://xxx.azurewebsites.net/solr/sitecore_master_index/admin/ping

3
  • Can you please share the script?
    – Astle
    Dec 8, 2022 at 9:11
  • 1
    @Astle see my updated answer above where I share detailed steps in my blog on how to setup an Azure Runbook with Powershell that auto-restarts Solr if there are any issues. Jun 9, 2023 at 8:34
  • Also worth noting - if you are running in a master/slave setup - setting the index as read-only via this workaround in SolrConfig.xml helps <!-- Bogus update handler - effectively makes index read-only --> <requestHandler name="/update" class="solr.SearchHandler" />
    – BikerP
    yesterday
0

We have faced the same issue when we hosted the SOLR as an app service.

We have simply changed the lockType: native to lockType: single

1
  • Sitecore support also recommends us to do this. How it goes so far?
    – purnadika
    May 23, 2023 at 2:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.