Requests stop being served due to Lucene Deadlock

Question

I'm currently running into an issue in production Content Delivery where every now and then (2 hours to 2 days), all requests will stop being served. Via memory dump we've been able to track down the cause as a thread deadlock - always a piece of code that is querying a Lucene index. My understanding of Lucene tells me that, it would seem unlikely that a read operation against Lucene would cause a deadlock but I'm not sure what would cause one.

Answer 1

We had a similar problem in a production installation that had a significant number of calculated Lucene fields, plus a number of very active content authors with a lot of publishing and index updating activity.

Essentially Sitecore was pushing events at Lucene faster than it could process them.

Here are some suggestions:

Try switching your index update strategies to be more batch-oriented.

Here's an example of how to do this for the master index: Replace:

<strategy ref="contentSearch/indexConfigurations/indexUpdateStrategies/syncMaster" />

With:

<strategy ref="contentSearch/indexConfigurations/indexUpdateStrategies/intervalAsyncMaster" />

in your Website\App_Config\Include\Sitecore.ContentSearch.Lucene.Index.Master.config

Try dialing back the frequency of event queue polling

In sitecore.config modify this setting:

<eventing defaultProvider="sitecore">
  <providers>
    ...
  </providers>
  <eventQueue>
    <!-- Time between checking the queue for newly queued events. If new events are found they will be raised. -->
   <processingInterval>00:00:02</processingInterval>
  </eventQueue>
</eventing>

You can also set the index rebuild threshold to prevent Lucene from rebuilding your indexes too often, if you have a lot of activity and a large number of index entries:

<settings>
  <!-- Default value is 100,000 -->
  <setting name="ContentSearch.FullRebuildItemCountThreshold" value="100000" />
</settings>

This really helped us with some mysterious index syncing problems between servers. It took weeks with Sitecore Support to reach these conclusions. Hope it works for you too.

Answer 2

Are you using SwitchOnRebuildLuceneIndex instead of the default? That would at least rule out index writes competing with reads.

I think I worked on the project referenced by csulham . . . the customer had added some heavy post publish logic that ran on Sitecore's heartbeat thread. After the Sitecore publish, the onPublishEndAsync would kick-off some additional work with bundling css and js resources. This monopolized the heartbeat thread and prevented EventQueue and other important operations from running. I don't think this sounds like your issue, but maybe you've customized publishing in some fashion?