12

On the CM server, all jobs are getting queued, after the following jobs appear in a running state:

Running jobs before queue explodes

UpdateIndex_filesystem                              Indexing    Running False   08.11.2016 15:43:19 0   
UpdateIndex_web2                                    Indexing    Running False   08.11.2016 15:43:19 0   
UpdateIndex_web                                     Indexing    Running False   08.11.2016 15:43:19 0   
UpdateIndex_master                                  Indexing    Running False   08.11.2016 15:43:19 0   
Sitecore.Tasks.TaskDatabaseAgent                    schedule    Running False   08.11.2016 15:48:13 0   
UpdateIndex_filesystem                              Indexing    Running False   08.11.2016 15:43:19 0   
UpdateIndex_core                                    Indexing    Running False   08.11.2016 15:43:19 0
Index_Update_IndexName=sitecore_analytics_index     Indexing    Running False   08.11.2016 15:40:18 0   

After an AppPool recycle it takes about 15 to 30 minutes until the same problem happens again. The Sitecore log- and crawling.log-file, as well as the Solr logging monitor (/solr/#/~logging) are not giving any indication of an error.

Because it is always the sitecore_analytics_index update job, which is getting started first, i assume it could be an issue with a corrupt solr index. But the index looks good in the Solr Core Admin. It could also be anything with the UpdateIndex_filesystem, which i don't know where this one comes from.

I already checked permission issues with Process Monitor according to the following post, but nothing helped: Sitecore index update jobs getting stuck

Has anyone experienced the same issue? Any recommendations on further investigation are very welcome.

Update: Edited 14/11/16 10:38 am.

Here is what i already did, during investigation:

  • Changed syncMaster index update strategy to intervalAsyncMaster for all custom master indexes and sitecore master indexes
  • Set the Indexing.UpdateInterval setting's value to "00:00:00" to disable updates for the legacy search indexes
  • Removed the <exclude hint="list:AddExcludedTemplate"> node from the defaultSolrIndexConfiguration, which lead to the crawling log warnings: WARN You have specified both IncludeTemplates and ExcludeTemplates. This logic is not supported. Exclude templates will be ignored.
  • Set the intervalAsyncMaster update strategy's interval to "00:01:00"
  • Set the intervalAsyncMaster update strategy's <CheckForThreshold> to false
  • Added a for each index with a SolrDocumentSizeCommitPolicy of maximum 1MB
  • Removed index update strategies in SwitchMasterToWeb.config for the CD environments
  • Cleaned the Sitecore_master DB's EventQueue and validated that the "Sitecore.Tasks.CleanupEventQueue" agent's IntervalToKeep is set (01:30:00).
  • Truncated the Sitecore_master DB's PublishQueue and set "Sitecore.Tasks.CleanupPublishQueue" agent's property DaysToKeep to 2 days.

Sadly, we're still facing the issue. The running job queue is a lot shorter than before, because of the new index update strategies, but the indexing jobs are still getting stuck in a running state and are blocking the job queue.

Maybe there is a relation to new added Languages. We went up from 19 Languages to 39. This means a lot more items are getting indexed. All Solr cores together are using ~11GB disk space. Solr Logging isn't showing any errors or warnings and Dashboard is looking fine as well:

  • Physical Memory: 3.85 / 32 GB (12%)
  • Swap Space: 4.8 / 36.75 GB (11.1%)
  • JVM-Memory: 781.83 MB / 1.92 GB (39.8%)

Solr looks healthy to me.

Update: Edited 24/11/16 11:20 am.

I finally solved the issue with a new setup of the website including physically web root, IIS-website and application pool as well as setting the AppPool permissions. I suspect wrong configured AppPool user permissions on some files in the web root, which could have happened during the manual deployment process or a broken AppPool with corrupted userprofiles/accounts.

Conclusion: Using automated deployments is going to avoid such issues - saving time and nerves. At least the Solr indexing configurations have been improved ;)

3
  • 2
    try increasing the crawling logs log level to debug in the config. This helped me understand what exactly was going on with the index when I had the issue with Jobs stuck. Perhaps add some of that info to the ticket then if there's something interesting.
    – Ian Graham
    Nov 8, 2016 at 17:38
  • can you please check SITECORE_ANALYTICS_INDEX_....._LAST_UPDATED setting value (table row value) in your sitecore instance Core database Properties table ? Nov 11, 2016 at 11:34
  • There is also a indexing backup bug in Sitecore 8.0. Let me know if you need more details on that.
    – Gabbar
    Nov 27, 2016 at 21:16

2 Answers 2

3

I just wanted to add this for how I solve the security issues @guschi had.

Download Process Monitor from the Microsoft web site.

When you get it open, add the rule Result - Contains - access. This will only capture access denied hits. Eliminating all the other chatter from the drives. Be sure to click the Add button.

enter image description here

Then with the app open, uncheck the boxes for registry, network and services. This will leave only file system turned on.

enter image description here

Now when you try and run your index, you should see the access denieds piling up. If you open one, on the second tab you can see the user and the service that is causing the issue.

enter image description here

2

Check your Event Queue in Master and Web databases.

It surely would have thousands of records. When there are a lot of event queue entries, the ManagedPoolThreads will get busy with background jobs and other jobs like indexing will be hung up.

Good to take a full backup of your databases, truncate the EventQueue table and check if it works.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.