Creating a new index in Azure Search for Sitecore 9 Helix structured

Question

Background: I am completely new to Azure Search in general. New to SOLR, Lucene the whole thing. I have a good bit of experience in SQL design for indexes and such, but this seems completely different.

So, I need to create a new search index for certain templates in our Sitecore 9 installation. Our site was originally developed by a 3rd party so I am coming into this project late.

What I need to do is, create an index to be used for 'news articles'. Which are a certain type of template in SC.

In the existing indexes, there are a LOT of fields, some have several hundred fields. I have some questions about that:

1. Is this normal to have that many fields for a Sitecore index?
2. How does one normally create a new index in Azure Search for Sitecore? What is the process?

I haven't been able to find any good examples of how to do this for Sitecore. How to set it up, sure, not how to actually create a usable index. In the Helix-based solution, there are several 'Feature' projects and most of them have their own configs for their specific indexes, that correspond to indexes that show up in SC admin 'Index Manager'. They also have xyz.ExcludeFields.config files and xyz.ExcludeTemplates.config files that list the majority of fields in the corresponding index to be excluded...

Judging by the sheer number of fields in the index, and these exclude files, I am guessing this can be auto-generated? Possibly?

Any insight to this would be welcomed. Thanks for reading.

Answer 1

Is it normal to have that hundreds of fields for a Sitecore index?

Yes - back in the days of Lucene and now Solr, some large multisite / multi language installations might see thousands of fields on the sitecore_master_index. This of course has always been a problem in terms of performance - the less fields on an index the better the performance. Lucene/Solr don't have a field limit, however as per the Solr performance guide - the number of fields is a big indicator of performance.

The full text search engine in Azure Search is built on Apache Lucene, and Microsoft in their wisdom decided that a 1000 field limit per index is "more than enough", which if you think about what you generally use an index for - a quick lookup based on some field values - you definitely shouldn't need that many fields to do what you need.

The core problem we are seeing in Sitecore 9.0 and 9.1 is that Sitecore have let the default Lucene/Solr setting of Index All Fields to true on Azure Search. There is probably a few reasons I can think of - backwards compatibility, core functionality that needs it, but I have heard on the grapevine that from Sitecore 9.2 it will be Index All Fields is false by default.

A whitelist approach when it comes to indexes in general is the best approach, rather than "index everything except this tediously built list of fields to exclude".

Creating a Helix compliant Azure Search Index

Using your example of needing to search for "News Articles" - we can create a custom index for this relatively easily. Firstly I will point you to this blog of Azure Search design considerations as I consider it best practice.

1. Index Configuration

First you need to decide on whether to use the defaultCloudIndexConfiguration (found in App_Config\Sitecore\ContentSearch.Azure\Sitecore.ContentSearch.Azure.DefaultIndexConfiguration.config).

Some considerations:

• You can use the defaultCloudIndexConfiguration in your custom index if you are going to change defaultCloudIndexConfiguration\documentOptions\indexAllFields to false. This is supported by Sitecore however if you are on 9.0.2 you will need to patch in a couple extra fields to get Experience Forms to work.
• Instead I would recommend to create a new CloudIndexConfiguration and reference parts of the defaultCloudIndexConfiguration that you are not changing.

So now you can create a new config file that will hold your CloudIndexConfiguration as well as your actual index definitions. To be helix compliant I would suggest creating it in your NewsArticles Feature project e.g. App_Config\Include\Feature\Feature.NewsArticles.ContentSearch.Azure.config

A basic starting point for the structure:

<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/" xmlns:role="http://www.sitecore.net/xmlconfig/role/" xmlns:search="http://www.sitecore.net/xmlconfig/search/">
  <sitecore role:require="Standalone or ContentDelivery or ContentManagement" search:require="Azure">
    <contentSearch>
      <indexConfigurations>
        <newsArticlesIndexConfiguration type="Sitecore.ContentSearch.Azure.CloudIndexConfiguration, Sitecore.ContentSearch.Azure">

          <initializeOnAdd>true</initializeOnAdd>
          <fieldMap             type="Sitecore.ContentSearch.Azure.FieldMaps.CloudFieldMap, Sitecore.ContentSearch.Azure">
            <fieldNames hint="raw:AddFieldByFieldName">
          <field fieldName="azureuniqueid"        cloudFieldName="azureuniqueid"      searchable="YES"  retrievable="YES"  facetable="YES"  filterable="YES"  sortable="YES"  boost="1f" type="System.String"   settingType="Sitecore.ContentSearch.Azure.CloudSearchFieldConfiguration, Sitecore.ContentSearch.Azure" />
          <field fieldName="_content"             cloudFieldName="content_1"          searchable="YES"  retrievable="NO"   facetable="NO"  filterable="NO"  sortable="NO"  boost="1f" type="System.String"   settingType="Sitecore.ContentSearch.Azure.CloudSearchFieldConfiguration, Sitecore.ContentSearch.Azure" />

...

      <documentOptions type="Sitecore.ContentSearch.Azure.CloudSearchDocumentBuilderOptions,Sitecore.ContentSearch.Azure">
        <indexAllFields>false</indexAllFields>

The important part here is indexAllFields is false, so you need to then build up your whitelist of fields. This has so many performance and maintenance benefits.

Here is a good little SPE script that helps to find what fields exist on a template. This should make it easier to build your white list.

An example of referencing (inheriting) the defaultCloudIndexConfiguration as follows:

<virtualFields ref="contentSearch/indexConfigurations/defaultCloudIndexConfiguration/virtualFields" />

2. Index Definitions

Once you have an index configuration you can then add your indexes to use it.

<indexes hint="list:AddIndex">
  <index id="newsarticle_web_index" type="Sitecore.ContentSearch.Azure.CloudSearchProviderIndex, Sitecore.ContentSearch.Azure">
    <param desc="name">$(id)</param>
    <param desc="connectionStringName">cloud.search</param>
    <param desc="totalParallelServices">5</param>
    <!-- This initializes index property store. Id has to be set to the index id -->
    <param desc="propertyStore" ref="contentSearch/indexConfigurations/databasePropertyStore" param1="$(id)" />
    <configuration ref="contentSearch/indexConfigurations/newsArticleIndexConfiguration" />
    <schemaBuilder ref="contentSearch/searchServiceSchemaBuilder" />
    <searchService type="Sitecore.ContentSearch.Azure.Http.CompositeSearchService, Sitecore.ContentSearch.Azure" />
    <strategies hint="list:AddStrategy">
      <!-- NOTE: order of these is controls the execution order -->
      <strategy role:require="Standalone OR ContentManagement" ref="contentSearch/indexConfigurations/indexUpdateStrategies/onPublishEndAsync" />
      <strategy role:require="ContentDelivery" ref="contentSearch/indexConfigurations/indexUpdateStrategies/manual" />
    </strategies>
    <commitPolicyExecutor type="Sitecore.ContentSearch.CommitPolicyExecutor, Sitecore.ContentSearch">
      <policies hint="list:AddCommitPolicy">
        <policy type="Sitecore.ContentSearch.ModificationCountCommitPolicy, Sitecore.ContentSearch">
          <Limit>1000</Limit>
        </policy>
      </policies>
    </commitPolicyExecutor>
    <locations hint="list:AddCrawler">
      <crawler type="Sitecore.ContentSearch.SitecoreItemCrawler, Sitecore.ContentSearch">
        <Database>web</Database>
        <Root>/sitecore/content/Site/Home</Root>
      </crawler>
    </locations>
    <enableItemLanguageFallback>false</enableItemLanguageFallback>
    <enableFieldLanguageFallback>false</enableFieldLanguageFallback>
  </index>

I've just done an example there for the Web database, but you can just copy it for Master and be sure to change the indexUpdateStrategies to syncMaster.

Known issues with Azure Search

• As mentioned - having the default indexAllFields=true is an issue and can be fairly easily rectified. If you're on 9.0 you will need this patch to fix Experience Forms.
• In Sitecore 9.0 there is no support for indexing PDFs so you will need this patch.
• In Sitecore 9.0 and 9.1 there is some less-than-desirable issues around default fields hitting a 32kB field size limit. Additionally the default submit batch size is not optimised. This community code can be patched into your solution to rectify this.
• In Sitecore 9.0 and 9.1 the ContentDelivery role has index updates enabled. I have raised a support ticket and Sitecore has confirmed it should be disabled. You can use this example patch to disable it.
• Sitecore 9.0.2 - logs get spammed with warnings - use this patch to fix.

Conclusion

And that's it! To round off - trying to script up an exclusion list is a losing battle. It's easier and much more beneficial to instead create a white list of included fields.