0

I know that I can configure my custom crawler (FlatDataCrawler) with manual strategy, and then I can start Rebuilding manually.

But how can I configure my crawler to start Refreshing every specific interval - let's say, every 5 minutes? But without removing all indexed items on start.

Goal: I want to index a huge database and I do not want the index to be cleaned, but only refreshed. Unfortunately, the rebuild always deletes everything first.

My current solution:

  1. Instance of FlatDataCrawler

  2. Manual strategy

  3. Task, which runs Refresh every 5 minutes.

Ja it possible to do it without my custom Task?

  • I think you should add some more context. What is your data? Where is it coming from? How often does it change? Etc... – Marek Musielak Jun 5 '19 at 14:22
  • I'm just asking how to configure crawler to run refreshing without rebuild action, witch clears all existing items on start. I can do the rest on my own. Thanks – g_m Jun 5 '19 at 14:32
  • You can't. That's why I'm asking what is your goal so maybe one can think about a workaround or better solution. – Marek Musielak Jun 5 '19 at 14:38
  • How does your database get updated? Start there, you would only want to update your index when the source is updated. Don't do it on a scheduled task. – Richard Seal Jun 5 '19 at 17:00
  • Since my datasource is external webservice, I never know what and when has chaned. That's why I'm using Tasks. – g_m Jun 5 '19 at 17:31
1

Refresh Strategy

Sitecore does provide strategy TimedIndexRefreshStrategy to refresh an index. It uses an interval based trigger.

In Sitecore 8.* it is used by the sitecore_analytics_index index.

Sitecore 9 does not use it at all and you won’t be able to find it in Sitecore config files.

If the strategy matches you need, you can add it to your custom index, for example, in the following way:

<index id="my_custom_index" … >
   …
   <strategies hint="list:AddStrategy">
      <timed type="Sitecore.ContentSearch.Maintenance.Strategies.TimedIndexRefreshStrategy, Sitecore.ContentSearch">
         <param desc="interval">00:05:00</param>
      </timed>
   </strategies>
   …
</index>

However you should also remember that the refreshing approach has a major disadvantage. It is simply not able to track deleted items. Refreshing always goes though existing items. If an item was deleted from your data source, refreshing will never reach it.

This is why I suggest you following the Switch On Rebuild solution


Switch On Rebuild

By default Sitecore does delete content from an index when it is being rebuilt. During that time search requests do not return any data.

Nevertheless Sitecore provides a solution for this issue. Its main idea is to switch solr cores on rebuild.

Switch On Rebuild in Sitecore or SWAP in Solr are out of the box features.

You can set up Solr to rebuild an index in a separate core so that the rebuilding does not affect the search index that is currently used. Once the rebuilding and the optimization of the index completes, Sitecore switches the two cores, and the rebuilt and optimized index is used.

You have to duplicate solr cores and apply some configuration changes to enable it for your indexes. Follow this guide to archive it.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.