Rebuilding of xDB index doesn't start after scaling up Azure Search

Question

We have an issue with Sitecore xDB index in Azure Search on Sitecore 9.1.1.

• Recently we tried to rebuild xDB index but we ran out of storage - 25GB was not enough to keep xDB index, secondary xDB index for rebuilding and default Sitecore indexes.
• We scaled up Azure Search by adding one more partition - now we have 50GB.
• We restarted xc-search app service and started rebuilding of xDB index again.
• Secondary index was removed and recreated, logs in IndexWorker contain only standard message

[Information] Starting Xdb Indexer App Domain, Machine: DW1MDWK000006, InstanceName: XConnectIndexer_AppDomain [Information] AppInsightsPerformanceCounters Constructor, Instance:XConnectIndexer, Path: App_Data\Diagnostics, CounterFilePattern: *.json [Information] Done.

• After 1 hour there was only 1 document in secondary index:

{"id": "indexconfiguration", "lastupdated": "2022-05-26T01:43:46.906Z" }.

• There were no errors in xc-search logs and no errors in IndexWorker logs.
• We restart xc-search app service again, executed .\XConnectSearchIndexer.exe -rr again and same story - secondary index removed and recreated, after few minutes 1 document in the index. Same as earlier. Nothing more.
• We let IndexWorker to process over night but no change. 1 document in the index.
• I tried again in the morning, same story. Index is being removed from Azure Search and new empty one is being created. But then, only 1 document is there after few minutes {"id": "indexconfiguration", "lastupdated": "2022-05-26T01:43:46.906Z", "isactivecore": false } and nothing more

When I run $filter=id eq 'xdb-rebuild-status'&$select=lastupdated,id,rebuildstate on primary index, I see:

{
  "@search.score": 1,
  "id": "xdb-rebuild-status",
  "lastupdated": "2023-10-03T05:40:59.243Z",
  "rebuildstate": 2
}

Last updated matches the time when I triggered rebuild in the morning, rebuildstate: 2 stands for starting.

Does anyone have any idea why IndexWorker does not rebuild xDB index after scaling up Azure Search?

Answer 1

I think it is possible that the old 'rebuild document' is still present on your primary core, from your previous failed attempt. Because of this, the process is failing to start, as it believes it is already in a started state.

To remove the document from Azure Search please try issuing a POST request to the following URI with the appropriate service name, index name and api admin key (as a header) filled in:

POST https://[service name].search.windows.net/indexes/[index name]/docs/index?api-version=2017-11-11  
Content-Type: application/json   
api-key: [admin key]  

With the following request body:

{  
  "value": [  
    {  
      "@search.action": "delete",  
      "id": "xdb-rebuild-status"  
    }  
  ]  
}

Then try restarting the rebuild process.

Credit for the deleting azure document here: https://stackoverflow.com/questions/54339348/how-to-delete-specific-document-using-id-on-azure-search-rest-api