We just updated from Sitecore 8.2 Update 4 to Update 7 and we've started seeing some strange behavior when publishing new items.

The issue presents itself in that it takes awhile for newly created items take awhile to show up on the site after they are published. It turns out that they publish just fine, but it takes awhile for them to be indexed so that the Item Resolver can find them.

After turning on a bunch of logging, I have determined that when you publish, the Index_Update_IndexName=sitecore_web_index job runs right away as it should. If you publish JUST the new item, the job runs quickly and finishes quickly and the item will display.

If, however, you publish related item, the update job takes quite awhile to finish and the new item doesn't show up until the job finishes. This is true EVEN if you are doing a smart publish and only a couple items are updated.

In my testing I am working on a test environment that does not have much use. When I smart publish, only 3 items are updated, but it takes 6-7 minutes and my crawling log shows nearly 3000 items indexed following that one publish.

I'm not sure where to start troubleshooting this. The publish works, the index job kicks off as expected. There are no relevant errors logged (as far as I can tell). The indexing job finishes normally and the content is properly indexed after the job finishes running. It just seems like the indexing job is processing many more items than are necessary based on what was actually published.

I've tried truncating the EventQueue and PublishingQueue to be sure there isn't any data hanging around and it has not made any difference.

We are using SOLR indexes. Does anyone know of anything else to look at?

UPDATE 1 (as requested in the comments): Update strategy is onPublishEndAsync and that strategy is configured with CheckForThreshold set to true.

There is no Indexing.FullRebuildItemCountThreshold (or any FullRebuildItemCountThreshold for that matter) defined in any config. According to the documentation, this value will default to 100,000.

UPDATE 2 (as requested in the comments): There are no custom publishing pipelines except a couple that log extra data for the current publish item. There are none that publish extra items.

As for the extra debugging, here is what I see after turning it on. The publish log shows:

6812 14:23:00 INFO  Finished [Publishing] - ProcessQueue in 6793 ms
6812 14:23:00 INFO  Publish Mode : SingleItem
6812 14:23:00 INFO  Created : 0
6812 14:23:00 INFO  Updated : 128
6812 14:23:00 INFO  Deleted : 0
6812 14:23:00 INFO  Skipped : 12

The crawling log shows the following (I've removed interleaved updates of other indexes to just include the one in question, sitecore_web_index):

9884 14:23:00 DEBUG [Index=sitecore_web_index] OnPublishEndAsynchronousStrategy executing.
9884 14:23:00 DEBUG [Index=sitecore_web_index] OnPublishEndAsynchronousStrategy: Batch for processing [9315374; 9315501] Count: 128
9884 14:23:00 DEBUG IndexCustodian. IncrementalUpdate triggered on index sitecore_web_index. Data=Count=128
10624 14:23:33 DEBUG [Index=sitecore_web_index] Documents sent to SOLR: 500 out of 500 - solr status: 0
10624 14:23:58 DEBUG [Index=sitecore_web_index] Documents sent to SOLR: 500 out of 500 - solr status: 0
ManagedPoolThread #5 14:24:22 DEBUG [Index=sitecore_web_index] Documents sent to SOLR: 500 out of 500 - solr status: 0
ManagedPoolThread #5 14:24:44 DEBUG [Index=sitecore_web_index] Documents sent to SOLR: 500 out of 500 - solr status: 0
4644 14:25:04 DEBUG [Index=sitecore_web_index] Documents sent to SOLR: 500 out of 500 - solr status: 0
10624 14:25:30 DEBUG [Index=sitecore_web_index] Documents sent to SOLR: 500 out of 500 - solr status: 0
ManagedPoolThread #5 14:25:43 DEBUG [Index=sitecore_web_index] Documents sent to SOLR: 320 out of 320 - solr status: 0
ManagedPoolThread #5 14:25:43 DEBUG [Index=sitecore_web_index] Committing: Add: 3320; Update:0; DeleteUnique: 85; DeleteGroup: 0
ManagedPoolThread #5 14:25:43 DEBUG [Index=sitecore_web_index] Committed - solr status 0
ManagedPoolThread #5 14:25:43 DEBUG IndexDependentHtmlCacheManager triggered.
ManagedPoolThread #5 14:25:43 DEBUG IndexDependentHtmlCacheManager done (processed sites: 16).

The header of the crawling job reports 128 items in the batch (which matches the number of items published), but the job actually ends up sending 3320 updates to SOLR. Many of the 128 items in the batch are related items (templates and layouts) but almost all of the 3320 items sent to SOLR are content which is why filtering the indexing to just the content items will not have much of an effect on the issue.

  • The thing is that for the default web index, most of the item will be indexed as the root is set from the /Content. What you can do is to create a custom index configuration file so that it indexes only items and fields that you require. This should decrease the time taken when indexes are being refreshed Oct 24 '18 at 19:48
  • I had considered this, but I have the logging adjusted to log the index updates being made and all of the items being logged are already in the /Content portion of the tree so it would not seem to be the root of my problem. Oct 24 '18 at 20:52
  • Is this only happening for the sitecore_web_index? Are other indexes fine?
    – Gabbar
    Oct 25 '18 at 14:50
  • Yes, this is only the web index. When I watch the Job Viewer and the logs, only the web index update job fires up and hangs around for awhile. Oct 25 '18 at 16:14
  • @BenStichter Since this is the web database index, can you check on a few settings and add them here? 1. <CheckForThreshold>true</CheckForThreshold> 2. Indexing.FullRebuildItemCountThreshold ? I have faced similar issues before and they turned out to be threshold related. For reference - doc.sitecore.net/sitecore_experience_platform/…
    – Gabbar
    Oct 26 '18 at 21:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.