Azure search not adding documents from custom index unless renamed

Question

I am having some odd issues with Azure Search in Sitecore 9.2.0, not indexing any documents from my custom index. Locally, when indexing to SOLR there are no problems at all.

Background:

We use SwitchOnRebuild and I can see that the index schema is created, but when the rebuild job finished in Sitecore, the index disappears from the Azure Search index overview. I assume it is because no documents have been added to the index.

Additionally, I have connected to the index from both my Azure test environment aswell as form my local development machine. I don't know if that is a problem.

Foundation.Search.Azure.Search.DefaultIndexConfiguration.config:

<?xml version="1.0" encoding="utf-8" ?>
<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/" xmlns:role="http://www.sitecore.net/xmlconfig/role/" xmlns:search="http://www.sitecore.net/xmlconfig/search/">
  <sitecore role:require="Standalone or ContentDelivery or ContentManagement" search:require="azure">
    <contentSearch>
      <indexConfigurations>
        <searchCloudIndexConfiguration ref="contentSearch/indexConfigurations/defaultCloudIndexConfiguration">
          <documentOptions type="Sitecore.ContentSearch.DocumentBuilderOptions, Sitecore.ContentSearch">
            <indexAllFields>false</indexAllFields>

            <!-- Excluded templates -->
            <exclude hint="list:AddExcludedTemplate">
              <BucketFolderTemplateId>{ADB6CA4F-03EF-4F47-B9AC-9CE2BA53FF97}</BucketFolderTemplateId>
            </exclude>

            <!-- Included templates -->
            <include hint="list:AddIncludedTemplate">
              <Frontpage>{CC8444F9-EF2D-43BC-9E08-84446CF253FB}</Frontpage>
              <StandardPage>{69E35420-AF97-40FC-AC84-E2F4AE256BEB}</StandardPage>
              <NewsArticlePage>{EFB6DB4E-4A31-413F-AD26-E808A52ED0EF}</NewsArticlePage>
            </include>

            <!-- GLOBALLY INCLUDE FIELDS IN INDEX
                 This setting allows you to specify which fields to include in the index when the indexAllFields setting is set to false.
            -->
            <include hint="list:AddIncludedField">
              <__Boost>{93D1B217-B8F4-462E-BABF-68298C9CE667}</__Boost>
              <__Created_By>{5DD74568-4D4B-44C1-B513-0AF5F4CDA34F}</__Created_By>
              <Date_Range>{7146F1A4-45FB-4CEC-9855-C95E9E595827}</Date_Range>
              <ID>{5A531AF0-C44C-4141-A0D3-09C5CDC3D654}</ID>
              <Image_Dimensions>{05EF282C-54DE-49B5-9EF3-0EB3008080C6}</Image_Dimensions>
              <Language>{BC06ED64-C4A1-4EE2-9835-541E1CC4CCC9}</Language>
              <Parent_ID>{1F4412CC-609C-4D3C-AF8C-D5C849202916}</Parent_ID>
              <Text>{A60ACD61-A6DB-4182-8329-C957982CEC74}</Text>
              <Title>{75577384-3C97-45DA-A847-81B00500E250}</Title>
              <Search_Types_Text>{E600C190-3F61-4776-B2F5-03AD7AEB351C}</Search_Types_Text>
              <Updated_Date>{87A830FB-4E2F-4F76-896B-F20CFA2374DD}</Updated_Date>
              <Workflow_State>{49D86313-493D-4054-ACC9-D68AD6B09332}</Workflow_State>
            </include>

            <!-- REMOVE INBUILT SITECORE FIELDS
                 This allows you to store a field in different ways in the index. You may want to store a field as Analyzed and Not Analyze
            -->
            <fields hint="raw:AddExcludedSpecialField">
              <field type="both">AllTemplates</field>
              <field type="both">Created</field>
              <field type="both">Editor</field>
              <field type="both">Hidden</field>
              <field type="both">Icon</field>
              <field type="both">Links</field>
              <field type="both">Updated</field>
            </fields>

            <!-- COMPUTED INDEX FIELDS
               This setting allows you to add fields to the index that contain values that are computed for the item that is being indexed.
               You can specify the storageType and indextype for each computed index field in the <fieldMap><fieldNames> section.
            -->
            <fields hint="raw:AddComputedIndexField">              
              <field fieldName="istemplate"                     >Sitecore.ContentSearch.ComputedFields.IsTemplate,Sitecore.ContentSearch</field>
              <field fieldName="haschildren"                    >Sitecore.ContentSearch.ComputedFields.HasChildren,Sitecore.ContentSearch</field>
              <field fieldName="lock"                           >Sitecore.ContentSearch.ComputedFields.IsLocked,Sitecore.ContentSearch</field>
              <field fieldName="culture"                        >Sitecore.ContentSearch.ComputedFields.Culture,Sitecore.ContentSearch</field>
              <field fieldName="site"                           >Sitecore.ContentSearch.ComputedFields.Site,Sitecore.ContentSearch</field>
              <field fieldName="version"                        >Sitecore.ContentSearch.ComputedFields.StoreVersionTermVector,Sitecore.ContentSearch</field>
              <field fieldName="urllink"                        >Sitecore.ContentSearch.ComputedFields.UrlLink,Sitecore.ContentSearch</field>
              <field fieldName="_content" returnType="string" type="Feature.Search.SubContentField, Feature.Search" />
              <field fieldName="renderedContent" returnType="string" type="Feature.Search.SubContentField, Feature.Search" />
            </fields>
          </documentOptions>
        </searchCloudIndexConfiguration>
      </indexConfigurations>
    </contentSearch>
  </sitecore>
</configuration>

Foundation.Azure.Search.Master.Index.config:

<?xml version="1.0" encoding="utf-8" ?>
<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/" xmlns:role="http://www.sitecore.net/xmlconfig/role/" xmlns:search="http://www.sitecore.net/xmlconfig/search/">
  <sitecore role:require="Standalone or ContentManagement" search:require="Azure">
    <contentSearch>
      <configuration type="Sitecore.ContentSearch.ContentSearchConfiguration, Sitecore.ContentSearch">
        <indexes hint="list:AddIndex">
          <index id="CUSTOMER_search_master_index" type="Sitecore.ContentSearch.Azure.CloudSearchProviderIndex, Sitecore.ContentSearch.Azure">
            <param desc="name">$(id)</param>
            <param desc="connectionStringName">cloud.search</param>
            <param desc="totalParallelServices">5</param>
            <!-- This initializes index property store. Id has to be set to the index id -->
            <param desc="propertyStore" ref="contentSearch/indexConfigurations/databasePropertyStore" param1="$(id)" />
            <configuration ref="contentSearch/indexConfigurations/searchCloudIndexConfiguration" >
              <scoringProfilesRepository ref="contentSearch/scoringProfilesRepository"/>
            </configuration>
            <schemaBuilder ref="contentSearch/searchServiceSchemaBuilder" />
            <searchService type="Sitecore.ContentSearch.Azure.Http.CompositeSearchService, Sitecore.ContentSearch.Azure"/>
            <strategies hint="list:AddStrategy">
              <!-- NOTE: order of these is controls the execution order -->
              <strategy ref="contentSearch/indexConfigurations/indexUpdateStrategies/manual" />
            </strategies>
            <commitPolicyExecutor type="Sitecore.ContentSearch.CommitPolicyExecutor, Sitecore.ContentSearch">
              <policies hint="list:AddCommitPolicy">
                <policy type="Sitecore.ContentSearch.TimeIntervalCommitPolicy, Sitecore.ContentSearch" />
                <policy type="Sitecore.ContentSearch.ModificationCountCommitPolicy, Sitecore.ContentSearch">
                  <Limit>1000</Limit>
                </policy>
              </policies>
            </commitPolicyExecutor>
            <locations hint="list:AddCrawler">
              <crawler type="Sitecore.ContentSearch.SitecoreItemCrawler, Sitecore.ContentSearch">
                <Database>master</Database>
                <Root>/sitecore/content/Web Sites</Root>
              </crawler>
            </locations>
            <enableItemLanguageFallback>false</enableItemLanguageFallback>
            <enableFieldLanguageFallback>false</enableFieldLanguageFallback>
          </index>
        </indexes>
      </configuration>
    </contentSearch>
  </sitecore>
</configuration>

What I have tried:

• Manually delete and rebuild the index
• Deleting entries referencing the customer-master-search-index in the properties table in the Core db
• Deleting and manually re-creating the index-catalog index in Azure Search

Observations

• Renaming the index (this actually works just fine: the new index is created and contains documents)
• If I disable, "SwitchOnRebuild" and rebuild my index, the document count is still 0. However, if I wait a couple of minutes (2-3) then my index appears with the correct number of documents (with the original name!).

Is it possible for an index to get into some sort of "bad state" which makes it impossible to rebuild? Or is Sitecore saving index state somewhere which I need to erase (like the index-catalog)?

Answer 1

It seems rubber-ducking with the community helped me solve the problem I have had for 3 days.

The solution: (it seems you have to do BOTH, although I am not 100% certain)

• Delete and manually re-create index-catalog
• Delete all references to the index in the properties table in the core database

I believe that the reason why I had the error was partly due to using "SwitchOnRebuild" and a corrupt index-catalog. In the index-catalog index, both the "primary" and the "secondary" index, was pointing to the same index in Azure Search and I think that confused Sitecore, so whenever I rebuilt my custom index, Sitecore switched to the secondary index and deleted the old primary index. However, since both the primary and the secondary was pointing to the same index, my custom index disappeared.