Is it normal to have that hundreds of fields for a Sitecore index?
Yes - back in the days of Lucene and now Solr, some large multisite / multi language installations might see thousands of fields on the sitecore_master_index. This of course has always been a problem in terms of performance - the less fields on an index the better the performance. Lucene/Solr don't have a field limit, however as per the Solr performance guide - the number of fields is a big indicator of performance.
The full text search engine in Azure Search is built on Apache Lucene, and Microsoft in their wisdom decided that a 1000 field limit per index is "more than enough", which if you think about what you generally use an index for - a quick lookup based on some field values - you definitely shouldn't need that many fields to do what you need.
The core problem we are seeing in Sitecore 9.0 and 9.1 is that Sitecore have let the default Lucene/Solr setting of Index All Fields to true on Azure Search. There is probably a few reasons I can think of - backwards compatibility, core functionality that needs it, but I have heard on the grapevine that from Sitecore 9.2 it will be Index All Fields is false by default.
A whitelist approach when it comes to indexes in general is the best approach, rather than "index everything except this tediously built list of fields to exclude".
Creating a Helix compliant Azure Search Index
Using your example of needing to search for "News Articles" - we can create a custom index for this relatively easily.
Firstly I will point you to this blog of Azure Search design considerations as I consider it best practice.
1. Index Configuration
First you need to decide on whether to use the
defaultCloudIndexConfiguration (found in
- You can use the
defaultCloudIndexConfiguration in your custom index if you are going to change
false. This is supported by Sitecore however if you are on 9.0.2 you will need to patch in a couple extra fields to get Experience Forms to work.
- Instead I would recommend to create a new
CloudIndexConfiguration and reference parts of the
defaultCloudIndexConfiguration that you are not changing.
So now you can create a new config file that will hold your
CloudIndexConfiguration as well as your actual index definitions. To be helix compliant I would suggest creating it in your NewsArticles Feature project e.g.
A basic starting point for the structure:
<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/" xmlns:role="http://www.sitecore.net/xmlconfig/role/" xmlns:search="http://www.sitecore.net/xmlconfig/search/">
<sitecore role:require="Standalone or ContentDelivery or ContentManagement" search:require="Azure">
<newsArticlesIndexConfiguration type="Sitecore.ContentSearch.Azure.CloudIndexConfiguration, Sitecore.ContentSearch.Azure">
<fieldMap type="Sitecore.ContentSearch.Azure.FieldMaps.CloudFieldMap, Sitecore.ContentSearch.Azure">
<field fieldName="azureuniqueid" cloudFieldName="azureuniqueid" searchable="YES" retrievable="YES" facetable="YES" filterable="YES" sortable="YES" boost="1f" type="System.String" settingType="Sitecore.ContentSearch.Azure.CloudSearchFieldConfiguration, Sitecore.ContentSearch.Azure" />
<field fieldName="_content" cloudFieldName="content_1" searchable="YES" retrievable="NO" facetable="NO" filterable="NO" sortable="NO" boost="1f" type="System.String" settingType="Sitecore.ContentSearch.Azure.CloudSearchFieldConfiguration, Sitecore.ContentSearch.Azure" />
The important part here is
indexAllFields is false, so you need to then build up your whitelist of fields. This has so many performance and maintenance benefits.
Here is a good little SPE script that helps to find what fields exist on a template. This should make it easier to build your white list.
An example of referencing (inheriting) the
defaultCloudIndexConfiguration as follows:
<virtualFields ref="contentSearch/indexConfigurations/defaultCloudIndexConfiguration/virtualFields" />
2. Index Definitions
Once you have an index configuration you can then add your indexes to use it.
<index id="newsarticle_web_index" type="Sitecore.ContentSearch.Azure.CloudSearchProviderIndex, Sitecore.ContentSearch.Azure">
<!-- This initializes index property store. Id has to be set to the index id -->
<param desc="propertyStore" ref="contentSearch/indexConfigurations/databasePropertyStore" param1="$(id)" />
<configuration ref="contentSearch/indexConfigurations/newsArticleIndexConfiguration" />
<schemaBuilder ref="contentSearch/searchServiceSchemaBuilder" />
<searchService type="Sitecore.ContentSearch.Azure.Http.CompositeSearchService, Sitecore.ContentSearch.Azure" />
<!-- NOTE: order of these is controls the execution order -->
<strategy role:require="Standalone OR ContentManagement" ref="contentSearch/indexConfigurations/indexUpdateStrategies/onPublishEndAsync" />
<strategy role:require="ContentDelivery" ref="contentSearch/indexConfigurations/indexUpdateStrategies/manual" />
<commitPolicyExecutor type="Sitecore.ContentSearch.CommitPolicyExecutor, Sitecore.ContentSearch">
<policy type="Sitecore.ContentSearch.ModificationCountCommitPolicy, Sitecore.ContentSearch">
<crawler type="Sitecore.ContentSearch.SitecoreItemCrawler, Sitecore.ContentSearch">
I've just done an example there for the Web database, but you can just copy it for Master and be sure to change the
Known issues with Azure Search
- As mentioned - having the default
indexAllFields=true is an issue and can be fairly easily rectified. If you're on 9.0 you will need this patch to fix Experience Forms.
- In Sitecore 9.0 there is no support for indexing PDFs so you will need this patch.
- In Sitecore 9.0 and 9.1 there is some less-than-desirable issues around default fields hitting a 32kB field size limit. Additionally the default submit batch size is not optimised. This community code can be patched into your solution to rectify this.
- In Sitecore 9.0 and 9.1 the ContentDelivery role has index updates enabled. I have raised a support ticket and Sitecore has confirmed it should be disabled. You can use this example patch to disable it.
- Sitecore 9.0.2 - logs get spammed with warnings - use this patch to fix.
And that's it! To round off - trying to script up an exclusion list is a losing battle. It's easier and much more beneficial to instead create a white list of included fields.