10

During an upgrade from 6.x to 8.1 we decided to also move the indexes from Lucene to Solr. We ran into quite some troubles, but that is not the question as those are fixed. The "problem" I'm still having is that an index rebuild for sitecore_web_index or sitecore_master_index is taking 5 to 6 hours (on a dedicated dev server). Maybe that is normal for 1 to 1.5M items.. maybe it will go faster on the prod environment. To get it running, we had to put 14GB of memory into the dev server as the initial 7 was not enough. We also had to increase the timeout on the SolrConnection a lot higher than the default (in the Autofac Solr setup).

We do use the index in code to search items based on path, template and or language but have no customizations to these indexes (we have a few custom indexes for those needs).

I was wondering however (this is the actual question) if there is a way to get the indexes rebuild faster. Some config changes to Sitecore or Solr maybe, or .. well anything that might help to reduce the time to rebuild.

Edit:

On the CD's I do not need the media items in the indexes and apparently I could disable those (which would help - a lot), but what would the side effects be on the CM? Will image fields still work correctly and will I be able to search for a media item? And how do I disable the media items correctly in a default solr config?

Improve this question
8
  • Four questions: first, have you looked at the "Logging" page of the Solr admin to see if you're getting any errors? Second, have you confirmed that this was not a problem before the upgrade, when you were still using Lucene? Third, you mentioned that you're index is rebuilding "on a dedicated dev server." Do you have both the Site and the Solr instances on the same server? If not, what is the connection speed like of the box that your Solr server is on. Fourth, do you have any Computed Index Fields? Are any of them new (i.e. you didn't haven them when you were on Lucene)?
    – Zachary Kniebel
    Commented Oct 24, 2016 at 15:53
  • 1. The Solr logs are clean - no errors. 3. 1 server with site and solr together 4. no computed fields in those indexes 2. the indexes with lucene took at least that long. I don't think there is actually a real "problem" in the setup - I just hope there are some tips to get it faster, surely there are other sites with 1M items?
    – Gatogordo
    Commented Oct 24, 2016 at 15:59
  • One more question: does is this server's Solr instance installed on a Solid-State Drive?
    – Zachary Kniebel
    Commented Oct 24, 2016 at 17:27
  • Can you try to disable indexing media items and check if it helps? I remember some document processing took ages in on of my past projects
    – Marek Musielak ♦
    Commented Oct 24, 2016 at 19:03
  • That could help as we have a lot of media items, but how do we that on the master/web indexes and what are the possible side effects of doing that?
    – Gatogordo
    Commented Oct 24, 2016 at 19:05

2 Answers 2

Reset to default
3

I answered a similar question with Coveo: Speeding up Coveo reindex

I am a Coveo guy but eh, indexing is indexing.

So as Zachary said in the comments, ensure you do not have any errors and be careful with Computed Fields. I would also add to be careful with anything touching the indexing pipeline, ex: Inbound Filters.

Solr is closer to Coveo than Lucene in it's possibility to scale. If you feel that your indexing is too slow, I would suggest to split your indexes using the crawler root:

https://stackoverflow.com/questions/38198334/sitecore-solr-index-configuration-set-different-root-path

This way you could rebuild your indexes without large media items. And maybe separate your media library in smaller pieces. You can also scale your Solr using slaves. @Søren Engel from Pentia did a great series of posts about scaling:

https://soen.ghost.io/tackling-the-challenges-of-architecting-a-search-indexing-infrastructure-in-sitecore-part-1/

Slaves would get rid of the query load on the indexing server.

Improve this answer
2
  • On the CD's I do not need the media items in the indexes which would help, but what would the side effects be on the CM? Will image fields still work correctly and will I be able to search for a media item?
    – Gatogordo
    Commented Oct 25, 2016 at 7:36
  • If you want to search for a media item, you will need to index it. The goal of using crawler roots is not to reduce total Indexing time, but to separate it in smaller rebuild. You could also only index your files, but ignore items like icons and thumbnails, which are references on the item, but not something that you want to be searched.
    – Simon Langevin
    Commented Oct 25, 2016 at 11:43
0

Check my answer to similar question - https://sitecore.stackexchange.com/a/1503/633

sitecore_web_index and sitecore_master_index stores all the fields, for all the versions for all the items by default.

Look at your implementation and decide if you need all that data there.

Improve this answer
4
  • On the CD's I do not need the media items in the indexes which would help, but what would the side effects be on the CM? Will image fields still work correctly and will I be able to search for a media item?
    – Gatogordo
    Commented Oct 25, 2016 at 7:36
  • @Gatogordo Media Search dialog uses Index, so Items for Media should be there, although blobs (image and pdf bytes) are not necessary.
    – Dmitry Harnitski
    Commented Oct 25, 2016 at 19:48
  • Is there a way to get only the blobs out?
    – Gatogordo
    Commented Oct 25, 2016 at 20:33
  • Blob is a computed field. Remove/update that field and you will get that.
    – Dmitry Harnitski
    Commented Oct 26, 2016 at 0:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.