Is Sitecore_Web_Index mandatory?

Question

I have a Sitecore (9.3) JSS based Multisite environment (10+ regional sites). Each site has a separate index, as well as the Sitecore Web index and Sitecore Master index.

The Sitecore Web index now contains more than 1 million documents, 20% of which are Sitecore content and 80% of which are media library items. As a result, the index rebuild takes longer, which affects the performance of index rebuild. Appreciate your inputs on the following.

• If not used in any searches, is it still necessary to index the media items (in Sitecore_web_index or Sitecore_master_index)?
• Given that we already have a distinct sitewide index, can you see any limitation on completely turning off this Sitecore_web_index?

Answer 1

I wouldn't remove any of the built-in indexes, as Sitecore (may) use them internally. For example if an editor switches to web in the Content Editor, you don't want potential exceptions in the UI.

But since you really don't care about the content of that index in this scenario, I'd change the crawler to index less content, such as just selected item paths or exclude just the Media Library. This will speed up crawling and reduce the size of you web index. Also consider setting <indexAllFields> to false unless you've already done so. Sometimes such small change would speed up things enough.

You can also change the indexing strategy to manual so that Sitecore doesn't spend resources on keeping the index up-to-date. Thereby you keep the index functionality and you can rebuild it your self if you ever need it in the future. It could for example be a scenario where you want to do a one-off carve among published data to build a report or something. Basically, everything works as default, but you'll have outdated data. There's almost always a small percentage of outdated data in the indexes anyway.

I'd also recommend you to poke around a bit in the Search.log files just to be sure you don't have any dependency on the web index. Just search for sitecore_web_index as the index name is always a part of the logged query.

Answer 2

I describe in this post about how to modify the indexes in Sitecore to crawl less data.

If you are not explicitly using the sitecore_web_index in your application then you should certainly reduce the amount of content crawled. It's probably less headache to do that than to completely remove it.

<index id="sitecore_web_index">
  <locations>
    <patch:delete />
  </locations>
  <locations hint="list:AddCrawler">
    <crawler name="templates" type="Sitecore.ContentSearch.SitecoreItemCrawler, Sitecore.ContentSearch">
      <Database>web</Database>
      <Root>/sitecore/templates</Root>
    </crawler>
    <crawler name="content" type="Sitecore.ContentSearch.SitecoreItemCrawler, Sitecore.ContentSearch">
      <Database>web</Database>
      <Root>/sitecore/content/home</Root>
    </crawler>
  </locations>
</index>