Azure Search index including too many fields

Question

Configuration: Sitecore 9.0.2, Azure PaaS

As most of us know by now, Azure Search has a hard limit on the number of fields that can be added to an index (1000). To resolve this issue in our environment, I developed the following patch file:

<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/" xmlns:role="http://www.sitecore.net/xmlconfig/role/" xmlns:search="http://www.sitecore.net/xmlconfig/search/">
    <sitecore role:require="Standalone or ContentDelivery or ContentManagement" search:require="Azure">
        <contentSearch>
            <configuration>
                <indexes>
                    <index id="sitecore_master_index">
                        <configuration>
                            <documentOptions ref="contentSearch/indexConfigurations/defaultCloudIndexConfiguration/documentOptions">
                                <!-- We only want default fields in the core/master/web indexes on azure due to 1000 field limit  -->
                                <indexAllFields>false</indexAllFields>
                                <!-- Fix sitecore 9 forms issue 229445 -->
                                <include hint="list:AddIncludedField">
                                    <isTemplate>{558F9307-EBAF-480D-88B5-DFE9E63A66DE}</isTemplate>
                                    <hidden>{39C4902E-9960-4469-AEEF-E878E9C8218F}</hidden>
                                </include>
                                <include hint="list:AddIncludedField">
                                    <my_custom_field>{EA795E6C-46E5-48B0-ACDA-E7288A018646}</my_custom_field>
                                    <!-- ... other fields omitted for brevity ... -->
                                </include>
                            </documentOptions>
                        </configuration>
                    </index>
                </indexes>
            </configuration>
        </contentSearch>
    </sitecore>
</configuration>

As you can see, I've disabled indexAllFields and explicitly included only those that we need.

However, when we run an index rebuild I'm still getting the below error in the logs:

ERROR [Index=sitecore_master_index] Commit failed

[...]

Nested Exception

Exception: Sitecore.ContentSearch.Azure.Http.Exceptions.AzureSearchServiceRESTCallException

Message: {"error":{"code":"","message":"The request is invalid. Details: definition : Invalid index: The index contains 1004 leaf fields (fields of a non-complex type). An index can have at most 1000 leaf fields.\r\n"}}

I looked in Azure and confirmed that my index has many other fields that I haven't included in my config. Why are these fields still being added?

Answer 1

Solution

Delete the existing Azure Search index (sitecore-master-index, in this case) before rebuilding the index.

Why?

The rebuild process does not recreate the index (at least not initially) so if your index was already built with many other fields, it won't clear those out before building it fresh.

How

If you don't have Switch On Rebuild enabled, you can simply navigate to that index in your Azure portal and delete it.

If you do have Switch On Rebuild enabled, in order to minimize impact to your application, you should delete the index that is currently set to be secondary (known as the "rebuild" index). Note: this does not mean simply deleting the sitecore-master-index-secondary index; that could be the active one.

To figure out the current "rebuild" index

1. Open an Azure Query Editor instance to your core db (*-core-db)
2. Run the below query:

   SELECT * FROM Properties
   WHERE [Key] LIKE '%sitecore_master%_cloud_rebuild_index_name'