On the CM server, all jobs are getting queued, after the following jobs appear in a running state:
UpdateIndex_filesystem Indexing Running False 08.11.2016 15:43:19 0
UpdateIndex_web2 Indexing Running False 08.11.2016 15:43:19 0
UpdateIndex_web Indexing Running False 08.11.2016 15:43:19 0
UpdateIndex_master Indexing Running False 08.11.2016 15:43:19 0
Sitecore.Tasks.TaskDatabaseAgent schedule Running False 08.11.2016 15:48:13 0
UpdateIndex_filesystem Indexing Running False 08.11.2016 15:43:19 0
UpdateIndex_core Indexing Running False 08.11.2016 15:43:19 0
Index_Update_IndexName=sitecore_analytics_index Indexing Running False 08.11.2016 15:40:18 0
After an AppPool recycle it takes about 15 to 30 minutes until the same problem happens again. The Sitecore log- and crawling.log-file, as well as the Solr logging monitor (/solr/#/~logging) are not giving any indication of an error.
Because it is always the sitecore_analytics_index update job, which is getting started first, i assume it could be an issue with a corrupt solr index. But the index looks good in the Solr Core Admin. It could also be anything with the UpdateIndex_filesystem, which i don't know where this one comes from.
I already checked permission issues with Process Monitor according to the following post, but nothing helped: Sitecore index update jobs getting stuck
Has anyone experienced the same issue? Any recommendations on further investigation are very welcome.
Update: Edited 14/11/16 10:38 am.
Here is what i already did, during investigation:
- Changed syncMaster index update strategy to intervalAsyncMaster for all custom master indexes and sitecore master indexes
- Set the Indexing.UpdateInterval setting's value to "00:00:00" to disable updates for the legacy search indexes
- Removed the
<exclude hint="list:AddExcludedTemplate">node from the defaultSolrIndexConfiguration, which lead to the crawling log warnings: WARN You have specified both IncludeTemplates and ExcludeTemplates. This logic is not supported. Exclude templates will be ignored. - Set the intervalAsyncMaster update strategy's interval to "00:01:00"
- Set the intervalAsyncMaster update strategy's
<CheckForThreshold>to false - Added a for each index with a SolrDocumentSizeCommitPolicy of maximum 1MB
- Removed index update strategies in SwitchMasterToWeb.config for the CD environments
- Cleaned the Sitecore_master DB's EventQueue and validated that the "Sitecore.Tasks.CleanupEventQueue" agent's IntervalToKeep is set (01:30:00).
- Truncated the Sitecore_master DB's PublishQueue and set "Sitecore.Tasks.CleanupPublishQueue" agent's property DaysToKeep to 2 days.
Sadly, we're still facing the issue. The running job queue is a lot shorter than before, because of the new index update strategies, but the indexing jobs are still getting stuck in a running state and are blocking the job queue.
Maybe there is a relation to new added Languages. We went up from 19 Languages to 39. This means a lot more items are getting indexed. All Solr cores together are using ~11GB disk space. Solr Logging isn't showing any errors or warnings and Dashboard is looking fine as well:
- Physical Memory: 3.85 / 32 GB (12%)
- Swap Space: 4.8 / 36.75 GB (11.1%)
- JVM-Memory: 781.83 MB / 1.92 GB (39.8%)
Solr looks healthy to me.
Update: Edited 24/11/16 11:20 am.
I finally solved the issue with a new setup of the website including physically web root, IIS-website and application pool as well as setting the AppPool permissions. I suspect wrong configured AppPool user permissions on some files in the web root, which could have happened during the manual deployment process or a broken AppPool with corrupted userprofiles/accounts.
Conclusion: Using automated deployments is going to avoid such issues - saving time and nerves. At least the Solr indexing configurations have been improved ;)
