When and why to create a custom index

Question

In my experience, the default sitecore_master_index and sitecore_web_index work pretty fast with Solr. We have around 1 million documents in the index and it returns a result in between 0 to 10ms. I can also rebuild part of the index using Rebuild Tree button on developer tab, so I don't have to worry that if some part of the index is broken I have to do a full rebuild (which takes a lot of time).

When and why is recommended to create a custom index? What are the best practices here?

Answer 1

I don't think it takes 10 ms when you have complex query with facets.

The most important reasons for creating a separate index are the following:

1. you can index only the items you need, speeding up (re-)indexation

2. you can index only the fields you really need, speeding up (re-)indexation

3. your index will only contain what you need, making it faster to search what you need
4. you can control when the index gets updated

Answer 2

Next to the reasons already mentioned by our Climber, I would add:

• possibility to tweak the index in all ways possible - you can go really far in this tweaking your fields in the index, or even using custom document options etc
• when in need of (several) ComputedFields, I normally start considering a custom index (so I don't need to include these in the full index)

but as your question already indicated: if you don't need them, don't use them (although I rather like them..)

Answer 3

Here are some pretty good posts on God index vs Domain indexes. https://soen.ghost.io/tackling-the-challenges-of-architecting-a-search-indexing-infrastructure-in-sitecore-part-2/ http://jockstothecore.com/indexing-patterns-in-sitecore/

As can see even sitecore has been adding domain indexes as versions of sitecore has progressed.

As mentioned when you start having computed fields, if you add these to the default indexes you'll slow down index rebuilds. As master index already indexes all the items/versions in the master database, this is normally the slowest index to rebuild. So you might end up making your overall rebuild time slower. Having smaller domain indexes means quicker overall rebuild time, assuming you can scale up/have enough cpu cores free, and configured sitecore indexing max threads to utilise them to rebuild indexes in parallel. Using Solr also helps with rebuild speed.

For us as well we have customised our domain indexes search to not swallow errors (and not return empty list) if SOLR is down, as well as add add a reduced connection time out on queries, and a circuit breaker on queries. Something we couldn't do with the default sitecore indexes, due to the way they are expected to behave.

Answer 4

An additional minor benefit is that it makes it easier to debug the index using a tool like Luke (http://www.getopt.org/luke/). For example, if you were developing a blog, you might have 100 documents in a custom index as opposed to a million in the Sitecore indexes. The custom index will be much easier to browse and debug (as it would have far fewer documents and potentially fewer fields) than the monstrous Sitecore indexes.