21

I'm currently running into an issue in production Content Delivery where every now and then (2 hours to 2 days), all requests will stop being served. Via memory dump we've been able to track down the cause as a thread deadlock - always a piece of code that is querying a Lucene index. My understanding of Lucene tells me that, it would seem unlikely that a read operation against Lucene would cause a deadlock but I'm not sure what would cause one.

  • Does the log contain anything of interest from when the issue occurs? – Kasper Gadensgaard Oct 4 '16 at 12:14
  • Which version of sitecore are you on? Do you have default lucene configuration? Or have you modified it? – Anicho Oct 4 '16 at 12:33
  • 1
    @sestocker do you still get a deadlock, when your custom index is not in the picture? – Anicho Oct 4 '16 at 13:36
  • 2
    Could it be files being locked by virus scanning software or something similar? – Owen Niblock Oct 4 '16 at 20:10
  • 1
    The deadlock happens in the code that queries your custom index, right? Please include a code sample of the offending code, as well as code and configuration for your implementation of the custom index (if you think it may potentially help). – Dmytro Shevchenko Oct 5 '16 at 11:04
22

We had a similar problem in a production installation that had a significant number of calculated Lucene fields, plus a number of very active content authors with a lot of publishing and index updating activity.

Essentially Sitecore was pushing events at Lucene faster than it could process them.

Here are some suggestions:

Try switching your index update strategies to be more batch-oriented.

Here's an example of how to do this for the master index: Replace:

<strategy ref="contentSearch/indexConfigurations/indexUpdateStrategies/syncMaster" />

With:

<strategy ref="contentSearch/indexConfigurations/indexUpdateStrategies/intervalAsyncMaster" />

in your Website\App_Config\Include\Sitecore.ContentSearch.Lucene.Index.Master.config

Try dialing back the frequency of event queue polling

In sitecore.config modify this setting:

<eventing defaultProvider="sitecore">
  <providers>
    ...
  </providers>
  <eventQueue>
    <!-- Time between checking the queue for newly queued events. If new events are found they will be raised. -->
   <processingInterval>00:00:02</processingInterval>
  </eventQueue>
</eventing>

You can also set the index rebuild threshold to prevent Lucene from rebuilding your indexes too often, if you have a lot of activity and a large number of index entries:

<settings>
  <!-- Default value is 100,000 -->
  <setting name="ContentSearch.FullRebuildItemCountThreshold" value="100000" />
</settings>

This really helped us with some mysterious index syncing problems between servers. It took weeks with Sitecore Support to reach these conclusions. Hope it works for you too.

| improve this answer | |
  • 1
    This is great advice – G Killian Oct 5 '16 at 13:20
  • I was just following this. Are there any other steps involved like enable History Table or Event Queues? I'm not sure if these are enabled OOTB in Sitecore 7.5 update-2 or Sitecore 8.x – Harsh Baid Oct 17 '16 at 9:55
  • Assuming it's a multi-server installation, yes, you need the Event queue running. – Sergeant Sitecore Oct 17 '16 at 14:18
1

Are you using SwitchOnRebuildLuceneIndex instead of the default? That would at least rule out index writes competing with reads.

I think I worked on the project referenced by csulham . . . the customer had added some heavy post publish logic that ran on Sitecore's heartbeat thread. After the Sitecore publish, the onPublishEndAsync would kick-off some additional work with bundling css and js resources. This monopolized the heartbeat thread and prevented EventQueue and other important operations from running. I don't think this sounds like your issue, but maybe you've customized publishing in some fashion?

| improve this answer | |
  • I agree that SwitchOnRebuildLuceneIndex can solve some locking problems, but I've had problems with that running against the web index - It's great for master though. – Sergeant Sitecore Oct 5 '16 at 13:22
  • Clarification: SwitchOnRebuildLuceneIndex worked for me on the CM server, for the master index. It caused index corruption when we put it on the web index on the CD server(s). I suspect Publish events flooded Lucene (per above). – Sergeant Sitecore Oct 5 '16 at 13:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.