Rebuilding sitecore_master_index: The property 'property_id_s_t_de' does not exist on type 'search.documentFields'

Question

When we rebuild our sitecore_master_index index, we usually get an error during the rebuild process and the logs show several instances of the below stack trace:

 692 12:50:11 ERROR [Index=sitecore_master_index] Commit failed
Exception: System.AggregateException
Message: One or more errors occurred.
Source: mscorlib
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at System.Threading.Tasks.Parallel.ForWorker[TLocal](Int32 fromInclusive, Int32 toExclusive, ParallelOptions parallelOptions, Action`1 body, Action`2 bodyWithState, Func`4 bodyWithLocal, Func`1 localInit, Action`1 localFinally)
   at System.Threading.Tasks.Parallel.ForEachWorker[TSource,TLocal](IEnumerable`1 source, ParallelOptions parallelOptions, Action`1 body, Action`2 bodyWithState, Action`3 bodyWithStateAndIndex, Func`4 bodyWithStateAndLocal, Func`5 bodyWithEverything, Func`1 localInit, Action`1 localFinally)
   at System.Threading.Tasks.Parallel.ForEach[TSource](IEnumerable`1 source, Action`1 body)
   at Sitecore.ContentSearch.Azure.Http.CompositeSearchService.PostDocuments(ICloudBatch batch)
   at Sitecore.ContentSearch.Azure.CloudSearchUpdateContext.Commit()

Nested Exception

Exception: Sitecore.ContentSearch.Azure.Http.Exceptions.BadRequestException
Message: Error in the request URI, headers, or body
Source: Sitecore.ContentSearch.Azure
   at Sitecore.ContentSearch.Azure.Http.SearchServiceClient.EnsureSuccessStatusCode(HttpResponseMessage response)
   at Sitecore.ContentSearch.Azure.Http.SearchServiceClient.PostDocuments(String jsonString)
   at Sitecore.Support.ContentSearch.Azure.Http.SearchService.PostDocuments(ICloudBatch batch) in C:\TeamCity\buildAgent3\temp\buildTmp\dbf4d98ab9a145aba55274027cfd1a07\src\Sitecore.Support.229598\ContentSearch\Azure\Http\SearchService.cs:line 76
   at Sitecore.ContentSearch.Azure.Http.CompositeSearchService.<>c__DisplayClass15_0.<PostDocuments>b__0(ISearchService searchService)
   at System.Threading.Tasks.Parallel.<>c__DisplayClass17_0`1.<ForWorker>b__1()
   at System.Threading.Tasks.Task.InnerInvokeWithArg(Task childTask)
   at System.Threading.Tasks.Task.<>c__DisplayClass176_0.<ExecuteSelfReplicating>b__0(Object )

Nested Exception

Exception: Sitecore.ContentSearch.Azure.Http.Exceptions.AzureSearchServiceRESTCallException
Message: {"error":{"code":"","message":"The request is invalid. Details: parameters : The property 'property_id_s_t_de' does not exist on type 'search.documentFields'. Make sure to only use property names that are defined by the type.\r\n"}}



 692 12:50:11 WARN  [Index=sitecore_master_index] Crawler : AddRecursive DoItemAdd failed - {AF26E057-F611-4C92-9B3C-CE85B1743632}
Exception: System.AggregateException
Message: One or more errors occurred.
Source: mscorlib
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at System.Threading.Tasks.Parallel.ForWorker[TLocal](Int32 fromInclusive, Int32 toExclusive, ParallelOptions parallelOptions, Action`1 body, Action`2 bodyWithState, Func`4 bodyWithLocal, Func`1 localInit, Action`1 localFinally)
   at System.Threading.Tasks.Parallel.ForEachWorker[TSource,TLocal](IEnumerable`1 source, ParallelOptions parallelOptions, Action`1 body, Action`2 bodyWithState, Action`3 bodyWithStateAndIndex, Func`4 bodyWithStateAndLocal, Func`5 bodyWithEverything, Func`1 localInit, Action`1 localFinally)
   at System.Threading.Tasks.Parallel.ForEach[TSource](IEnumerable`1 source, Action`1 body)
   at Sitecore.ContentSearch.Azure.Http.CompositeSearchService.PostDocuments(ICloudBatch batch)
   at Sitecore.ContentSearch.Azure.CloudSearchUpdateContext.Commit()
   at Sitecore.ContentSearch.CommitPolicyExecutor.IndexModified(IProviderUpdateContext context, Object document, IndexOperation operation)
   at Sitecore.ContentSearch.Azure.CloudSearchUpdateContext.AddDocument(Object itemToAdd, IExecutionContext[] executionContexts)
   at Sitecore.ContentSearch.Azure.CloudSearchIndexOperations.Add(IIndexable indexable, IProviderUpdateContext context, ProviderIndexConfiguration indexConfiguration)
   at Sitecore.ContentSearch.SitecoreItemCrawler.DoAdd(IProviderUpdateContext context, SitecoreIndexableItem indexable)
   at Sitecore.ContentSearch.HierarchicalDataCrawler`1.CrawlItem(T indexable, IProviderUpdateContext context, CrawlState`1 state)

Nested Exception

Exception: Sitecore.ContentSearch.Azure.Http.Exceptions.BadRequestException
Message: Error in the request URI, headers, or body
Source: Sitecore.ContentSearch.Azure
   at Sitecore.ContentSearch.Azure.Http.SearchServiceClient.EnsureSuccessStatusCode(HttpResponseMessage response)
   at Sitecore.ContentSearch.Azure.Http.SearchServiceClient.PostDocuments(String jsonString)
   at Sitecore.Support.ContentSearch.Azure.Http.SearchService.PostDocuments(ICloudBatch batch) in C:\TeamCity\buildAgent3\temp\buildTmp\dbf4d98ab9a145aba55274027cfd1a07\src\Sitecore.Support.229598\ContentSearch\Azure\Http\SearchService.cs:line 76
   at Sitecore.ContentSearch.Azure.Http.CompositeSearchService.<>c__DisplayClass15_0.<PostDocuments>b__0(ISearchService searchService)
   at System.Threading.Tasks.Parallel.<>c__DisplayClass17_0`1.<ForWorker>b__1()
   at System.Threading.Tasks.Task.InnerInvokeWithArg(Task childTask)
   at System.Threading.Tasks.Task.<>c__DisplayClass176_0.<ExecuteSelfReplicating>b__0(Object )

Nested Exception

Exception: Sitecore.ContentSearch.Azure.Http.Exceptions.AzureSearchServiceRESTCallException
Message: {"error":{"code":"","message":"The request is invalid. Details: parameters : The property 'property_id_s_t_de' does not exist on type 'search.documentFields'. Make sure to only use property names that are defined by the type.\r\n"}}

A few strange things to note:

1. Multiple fields have errors on every rebuild (each field usually more than once, too).
2. The fields in the errors differ on every index rebuild.
3. Every field in the errors is present in the index.
4. The fields are not present on the template of the items referenced by the DoItemAdd warning message (with the ID of the item).
5. The fields are always shared fields.
6. Though there are multiple instances of the error for each field name, most of the instances' IDs reference the same template for each field name.

What could be causing this issue?

Environment:

• Sitecore 9.0.2
• Azure PaaS
• Our Azure SearchService is from a Sitecore Support hotfix (229598)

Answer 1

Based on the latest information it looks like the issue may be related to a bug that was fixed in Sitecore 9.2 (release notes):

​​In Azure Search, the SwitchOnRebuildEventRemote event is shared through the EventQueue of the Web database instead of the system EventQueue.

To determine if this affects you, answer the two questions below:

1. Is the web database shared by CM and CD instance?
2. Does the issue affect the CD instance only?

If the answers are

1. No
2. Yes

there is a patch you can request from Sitecore Support to resolve the issue: 290638.

Note that this is a patch not a hotfix.

Answer 2

This can be related to the apiVersion specified in your cloud.search connection string.

Make sure it is set to the correct supported version for your version of Sitecore.

<add name="cloud.search" connectionString="serviceUrl=https://azure-search-resource.search.windows.net;apiVersion=2016-09-01;apiKey=__APIKEY__" />

We had to change ours from 2015-02-28-preview to 2016-09-01.

Important note

If you are using apiVersion=2016-09-01, you will also need to disable the ContentSearch.Azure.UseIsMatchScoring setting with the following patch.

<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
    <sitecore>
        <settings>
            <setting name="ContentSearch.Azure.UseIsMatchScoring">
                <patch:attribute name="value">false</patch:attribute>
            </setting>
        </settings>
    </sitecore>
</configuration>