I'm currently running into an issue in production Content Delivery where every now and then (2 hours to 2 days), all requests will stop being served. Via memory dump we've been able to track down the cause as a thread deadlock - always a piece of code that is querying a Lucene index. My understanding of Lucene tells me that, it would seem unlikely that a read operation against Lucene would cause a deadlock but I'm not sure what would cause one.
We had a similar problem in a production installation that had a significant number of calculated Lucene fields, plus a number of very active content authors with a lot of publishing and index updating activity.
Essentially Sitecore was pushing events at Lucene faster than it could process them.
Here are some suggestions:
Try switching your index update strategies to be more batch-oriented.
Here's an example of how to do this for the
<strategy ref="contentSearch/indexConfigurations/indexUpdateStrategies/syncMaster" />
<strategy ref="contentSearch/indexConfigurations/indexUpdateStrategies/intervalAsyncMaster" />
Try dialing back the frequency of event queue polling
sitecore.config modify this setting:
<eventing defaultProvider="sitecore"> <providers> ... </providers> <eventQueue> <!-- Time between checking the queue for newly queued events. If new events are found they will be raised. --> <processingInterval>00:00:02</processingInterval> </eventQueue> </eventing>
You can also set the index rebuild threshold to prevent Lucene from rebuilding your indexes too often, if you have a lot of activity and a large number of index entries:
<settings> <!-- Default value is 100,000 --> <setting name="ContentSearch.FullRebuildItemCountThreshold" value="100000" /> </settings>
This really helped us with some mysterious index syncing problems between servers. It took weeks with Sitecore Support to reach these conclusions. Hope it works for you too.
Are you using SwitchOnRebuildLuceneIndex instead of the default? That would at least rule out index writes competing with reads.
I think I worked on the project referenced by csulham . . . the customer had added some heavy post publish logic that ran on Sitecore's heartbeat thread. After the Sitecore publish, the onPublishEndAsync would kick-off some additional work with bundling css and js resources. This monopolized the heartbeat thread and prevented EventQueue and other important operations from running. I don't think this sounds like your issue, but maybe you've customized publishing in some fashion?