Sitecore 8: Items Duplicated in Web Index After Publish

Question

In only our Staging/Production environment, items are being duplicated in our web index after a publish is performed. The only distinction between the duplicate items (via Luke) is that any indexed images are prefixed with "/sitecore/shell"

If I perform a manual index rebuild from Indexing Manager, the results are as expected.

The index uses a copy of the default IndexConfiguration with some slight modifications (edited), renamed to be newsIndexConfiguration and it is referenced properly on the index implementation

<index id="$(News.Index.Name.Web)" type="Sitecore.ContentSearch.LuceneProvider.LuceneIndex, Sitecore.ContentSearch.LuceneProvider">
    <param desc="name">$(id)</param>
    <param desc="folder">$(id)</param>
    <!-- This initializes index property store. Id has to be set to the index id -->
    <param desc="propertyStore" ref="contentSearch/indexConfigurations/databasePropertyStore" param1="$(id)" />
    <configuration ref="contentSearch/indexConfigurations/newsIndexConfiguration" />
    <strategies hint="list:AddStrategy">
        <strategy ref="contentSearch/indexConfigurations/indexUpdateStrategies/onPublishEndAsync" />
    </strategies>
    <commitPolicyExecutor type="Sitecore.ContentSearch.CommitPolicyExecutor, Sitecore.ContentSearch">
        <policies hint="list:AddCommitPolicy">
            <policy type="Sitecore.ContentSearch.TimeIntervalCommitPolicy, Sitecore.ContentSearch" />
        </policies>
    </commitPolicyExecutor>
    <locations hint="list:AddCrawler">
        <crawler type="Sitecore.ContentSearch.SitecoreItemCrawler, Sitecore.ContentSearch">
            <Database>web</Database>
            <Root>/sitecore/content/Global/News/Articles</Root>
        </crawler>
    </locations>
</index>

It goes without saying that there is custom code all over this implementation at this point. However, this is a new feature that started with the default configuration. I cannot figure out what might be causing the duplicates to be added on publish.

It was erring without the <indexAllFields> element set to true, which I can live with, but certainly isn't ideal.

I should note, when I perform a Republish (including all children) at the newsroom root, it adds these multiple values, not just 1 additional duplicate, but 5 or even more sometimes, it feels quite arbitrary.

In the past, in this instance, I've gotten around this by filtering out results that contain "/sitecore/shell" in the image field, but this is merely a Band-Aid. Any help to solve this would be greatly appreciated.

Answer 1

This looks like a bug with the OnPublishEndAsync strategy.

It becomes apparent when authors start creating new versions of content each time they lock and edit an item.

It has been fixed in 8.1 but still resides in the version you are using.

See this knowledge base article

https://kb.sitecore.net/articles/992608

Answer 2

The issue was that in my efforts to trim the index of waste, i.e. remove fields that I'm not accessing, I removed a certain special field (though it is not marked in any way as being important, though by name it makes sense in hindsight). I removed the "_uniqueid" field from the index configuration. Specifically here:

<fieldMap>
...
  <field fieldName="_uniqueid"            storageType="YES" indexType="TOKENIZED"    vectorType="NO" boost="1f" type="System.String" settingType="Sitecore.ContentSearch.LuceneProvider.LuceneSearchFieldConfiguration, Sitecore.ContentSearch.LuceneProvider">
    <analyzer type="Sitecore.ContentSearch.LuceneProvider.Analyzers.LowerCaseKeywordAnalyzer, Sitecore.ContentSearch.LuceneProvider" />
  </field>
...

I had created a new IndexConfiguration.config file unique to my index. I tested and I was able to remove all other fields from the index. Toggling this field on or off reproduced the issue.

EDIT:

With the _uniqueid field omitted from the index configuration, the field will still appear in the index (via Luke or other means). Including it in the index configuration will solve this problem.

Answer 3

It may depend on what version of Sitecore you are running, from this SO post (https://stackoverflow.com/questions/27272832/urllink-in-sitecore-indexs-returns-media-url-with-sitecore-shell-media) it looks like there may be a known bug.

Accepted answer from link above:

Just to give an update for this. Confirmed by Sitecore it's a bug. They are finding a workaround, I'll update here when they do.

Thanks.

-------------updated on 15/12/2014----------------------

Ticket's closed now. Sitecore's solution is to remove this "urlLink" field from the index in the future(They said they will request this).

Reason being that url should be generated by LinkManager based on current site context. However site context doesn't exist while indexing(For content item you can check the path, but for media item you cannot).

Of course there's another solution is to create your own computeredField to override the logic for media item. However I agree with Sitecore, it feels more right to remove this field from the index.

On top of all that, the initial reason for using the urlLink field was to drop the need for querying Sitecore completely while doing a Search. All the content are coming from Index's stored fields value. But on the other hand, pagination is normally being used for search results, so even there're requests to Sitecore, it shouldn't be a lot.

That'd be all for now, happy to see more opinions and to discuss, thanks!