Customize Sitecore crawling logs to log request payload and request url on rebuild index

Question

I am facing some issues with index rebuild, Is there any way to Customize the Sitecore crawling logs to log the request payload and request URL on the rebuild index?

Which pipeline is responsible for logging details while index rebuilding?

Answer 1

A crawler is defined in the indexes configuration. It specifies a 'root' within the content tree. When the index is rebuilt, the crawlers recursively goes through the content tree from that root. See example below:

<locations hint="list:AddCrawler">
  <crawler type="Sitecore.ContentSearch.SitecoreItemCrawler, Sitecore.ContentSearch">
    <Database>web</Database>
    <Root>/sitecore/content/some/place/in/content/tree</Root>
  </crawler>
</locations>

Therefore there is no specifc request payload or url to log. Instead, a command is triggered, which ultimately calls CrawlTree (Sitecore.ContentSearch.HierarchicalDataCrawler).

private void CrawlTree(T indexable, IProviderUpdateContext context, CrawlState<T> crawlState)
{
    try
    {
        currentCrawlOperations[crawlState.Id] = crawlState;
        Interlocked.Increment(ref crawlState.PendingCrawlCount);
        CrawlItem(indexable, context, crawlState);
        while ((crawlState.PendingCrawlCount > 0 || crawlState.CurrentTasks.Count > 0) && !crawlState.IsCancelled && !crawlState.CancellationToken.IsCancellationRequested && crawlState.CurrentTasks.Count((KeyValuePair<IIndexableUniqueId, Task> task) => task.Value.IsFaulted) != crawlState.PendingCrawlCount)
        {
            Thread.Sleep(50);
        }
        CrawlingLog.Log.Info($"[Index={base.Index.Name}] Crawler: Total processed items {crawlState.CrawlCount}");
    }
    finally
    {
        currentCrawlOperations.Remove(crawlState.Id);
    }
}

Then in turn CrawlItem, which I guess is at a granual enough level that would allow you to target individual items.

private void CrawlItem(T indexable, IProviderUpdateContext context, CrawlState<T> crawlState)
{
...
}

In each of the above functions, you will see references to the CrawlingLog.Log.Info, which sends data to the CrawlingLogFileAppender and in turn to the Crawling logs you see in the sitecore App_Data/Logs folder.

CrawlingLog.Log.Info($"[Index={base.Index.Name}] Crawler: Processed {num} items");

UPDATE

If you are seeing duplicate results, then my advice would be to look first directly in the SOLR index. Run the exact query directly there and you will hopefully be able to see some more information about the duplicates (i.e. they might appear to be dupictae, but in fact have separate Item IDs).

To do this, follow these steps:

• Perform your search action on CD server
• Review the latest search log (i.e. App_Data/Logs/Search.log.20241009.151254)
• The last line in that log should relate to the above query. Copy everything starting ?q=
• Navigate to SOLR Dashboard (i.e. paste the value of solr.search conn string into browser on CD server)
• Use core selector to select core
• Click Query
• Paste the line from your search log at the end of the URL in browser

Once you follow the above steps, you should see the JSON response from SOLR directly. Look here to compare the duplciate results and compare the fields. Double check _uniqueid, versions, database etc. You may find some answers there.

Answer 2

In order to get detailed insights for troubleshooting purposes, you will need to change the logging level to DEBUG inside the crawling log.

<logger name="Sitecore.Diagnostics.Crawling" additivity="false">
      <level value="DEBUG" />
      <appender-ref ref="CrawlingLogFileAppender" />
</logger>

Moreover you can enable the verbose logging in order to get details on item not getting indexed or some fields getting skipped in the rebuild process.

<?xml version="1.0" encoding="utf-8" ?>
<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
  <sitecore>
    <settings>
      <!--  VERBOSE LOGGING
            This feature is designed to facilitate search index configuration and provide the necessary insight
            in troubleshooting scenarios. For example, if a particular item is not getting indexed,
            the VerboseLogger can provide more context and help you to figure out the problem.

            It is important that you only enable the VerboseLogger component in special circumstances
            and never run it for long periods in a production environment.
            Otherwise, this would result in an extremely large log file, which may have performance implications.
      -->
      <setting name="ContentSearch.VerboseLogging" value="true" />
    </settings>
  </sitecore>
</configuration>

Combining both of them will give you detailed insights on the item level index rebuild process inside the Crawling log file.

Hope this helps!!! Let me know in case you have any queries.