2

I have a custom computedfield in Sitecore 9.1.1.

I am getting the below error on CD server only as my web.config has:

<add key="role:define" value="ContentDelivery" />

Current CustomIndex config:

<contentSearch>
      <indexConfigurations>
        <extendedSolrIndexConfiguration type="Sitecore.ContentSearch.SolrProvider.SolrIndexConfiguration, Sitecore.ContentSearch.SolrProvider">
          <documentOptions type="Sitecore.ContentSearch.SolrProvider.SolrDocumentBuilderOptions, Sitecore.ContentSearch.SolrProvider" >
            <fields hint="raw:AddComputedIndexField">
              <field fieldName="helpdesk_content" storageType="NO" indexType="TOKENIZED">Sitecore.Feature.Helpdesk.Infrastructure.Fields.HelpDeskContentComputedField, Sitecore.Feature.Helpdesk</field>
            </fields>
          </documentOptions>
        </extendedSolrIndexConfiguration>
      </indexConfigurations>
    </contentSearch>

Custom Index code:

public object ComputeFieldValue(IIndexable indexable)
    {
      Assert.ArgumentNotNull(indexable, "indexable");
      string url = null;
      try
      {
        Item item = indexable as SitecoreIndexableItem;

        // This field only works for items uder /sitecore/content that have a layout
        if (item == null
            || item.Visualization.Layout == null
            || !item.Paths.FullPath.StartsWith(
                    Sitecore.Constants.ContentPath,
                    System.StringComparison.InvariantCultureIgnoreCase))
        {
          return null;
        }

        // Determine the url to request
        using (new DatabaseSwitcher(item.Database))
        {
          //explicitly resolve context site
          var site = item.GetContextSite();

          url = WebUtil.AddQueryString(
              LinkManager.GetItemUrl(item, new UrlOptions()
              {
                AlwaysIncludeServerUrl = true,
                Site=site
              }),
              "sc_database", Sitecore.Context.Database.Name);
        }

        // Http request the page
        using (var client = new WebClient())
        {
          Log.Debug(string.Format("Indexing content for the URL [{0}]", url));
          string pageContent = client.DownloadString(url);

          // Parse the page's html using HtmlAgilityPack
          HtmlDocument htmlDocument = new HtmlDocument();
          htmlDocument.LoadHtml(pageContent);

          // Strip out all the html tags, so we can index just the text
          HtmlNode mainContainer = htmlDocument.DocumentNode.Descendants("body").FirstOrDefault();
          string content = mainContainer != null ? GetAllInnerTexts(mainContainer) : null;
          Log.Debug("content indexed : " + content);
          return content;
        }
      }
      catch (WebException webExc)
      {
        Log.Warn(string.Format("Failed to html index {0} ({1}): {2}", indexable.Id, url, webExc.Message), webExc, this);
      }
      catch (Exception exc)
      {
        Log.Error(string.Format("An error occurred when indexing {0}: {1}", indexable.Id, exc.Message), exc, this);
      }
      return null;
    }

Error on CD server:

ERROR An error occurred when indexing {0CAA7DB1-CAF8-48EA-8BD0-DA191B4BCF46}: Could not find configuration node: databases/database[@id='core']
Exception: System.InvalidOperationException
Message: Could not find configuration node: databases/database[@id='core']
Source: Sitecore.Kernel
   at Sitecore.Configuration.DefaultFactory.GetConfigNode(String xpath, Boolean assert)
   at Sitecore.Configuration.DefaultFactory.CreateObject(String configPath, String[] parameters, Boolean assert)
   at Sitecore.Configuration.DefaultFactory.GetDatabase(String name, Boolean assert)
   at Sitecore.Configuration.DefaultFactory.GetDatabase(String name)
   at Sitecore.Sites.SiteContext.get_Database()
   at Sitecore.Foundation.Indexing.Infrastructure.Fields.HtmlCrawledField.ComputeFieldValue(IIndexable indexable) in C:\Projects\Unitywater\src\foundation\Indexing\code\Infrastructure\Fields\HtmlCrawledField.cs:line 51

Do I need to disable the custom indexing on the CD server on have to make it based on specific database? What sort of configuration you normally have in CD server for custom index?

3

As Mark Gibbons already wrote in his answer, you don't want your CD servers to run any indexing. All the work should be done by CM (or Standalone) servers.

Now why your code doesn't work? There may be multiple reasons here like:

  1. Sitecore.Context.Database.Name line. If onPublishEndAsync strategy is used, Sitecore.Context.Database will either throw exception or will be null or in best case it will be core. Use item.Database like you used for DatabaseSwitcher.
url = WebUtil.AddQueryString(..., "sc_database", item.Database.Name);
  1. You should think if you need to get content of the pages which are not in the web database. At the moment, if you have an item in core database and it's a SPEAK application with some presentation details on it, your code will run and try to make a request to that page. That's not ideal situation, isn't it? And I believe you don't need to get the content of items from master database neither, do you? I think it may be reasonable to only crawl the page if item.Database.Name == "web" or whatever the name of your database is.
if (item.Database.Name != "web")
    return null;
  1. Calling item.Visualization.Layout is rather heavy. It parses the Shared and Final Layout fields, applies XML deltas, etc. Maybe checking the raw string value would be enough:
if (string.IsNullOrEmpty(item[FieldIDs.LayoutField] + item[FieldIDs.FinalLayoutField]))
    return null;
  1. You may think about 2 approached to get the correct content page from the CM server. You can either has some config with CD server host name and apply it to the url or the item or crawl the item from CM server instead.

  2. Remember about personalization rules - not all variants of your content will be rendered on the pages.

  3. Think about the pages which are only available for logged in users. You don't want to crawl them. Otherwise, you may get let of login screen content in those pages.

  4. Your code ignores <title> and <meta name="description"> tags and they usually have very meaningful content. You may think about adding their values to the value of your computed field.

0

To answer your question, CD servers that use Solr or Azure Search should have indexing disabled, so that CM (or some other non-CD role) does the indexing.

  <index id="sitecore_web_index">
    <strategies hint="list:AddStrategy" patch:instead="strategies">
      <strategy role:require="Standalone OR ContentManagement" ref="contentSearch/indexConfigurations/indexUpdateStrategies/onPublishEndAsync"  />
      <strategy role:require="ContentDelivery" ref="contentSearch/indexConfigurations/indexUpdateStrategies/manual"/>
    </strategies>
  </index>

But mainly I would question what you're actually trying to do here - it seems a very inefficient way to index content. Much better would be to just directly get the field value(s) you need to index.

  • Does that config patch comes with sitecore by default, do I need to add that as a patch file so that sitecore doesn't index on ContentDelivery? I know it is very inefficient way of doing it but it is very specific to client requirements as they want to index data from external website which is injected into website. – Arun Kumar Jun 13 at 6:40
  • That patch is not included by default. – Mark Gibbons Jun 13 at 9:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.