8

I have thought about how to best assemble a "page" document in Lucene/Solr to later use in Sitecore when searching for most relevant pages to show to a user.

A page consist of many elements which is typically indexed separately in Sitecore.

How would I best assemble the item that represents a given "page"? As I see it I would need to extract all reference from the actual page item, look up all related items and store their content in the document representing the "page" document.

This seems kind of bulky if that needs to be performed each time a page item is changed.

Seems easier just to crawl all pages the old fashion way.

What is the best solution for indexing whole pages?

8

One approach that we have used is to index the content of all data sources in a single computed field. You can search that field when searching the index. This can probably be improved, but it's a start.

I believe this is also the approach the are using in the Habitat project.

Copied/inspired from: http://kamsar.net/index.php/2014/05/indexing-subcontent/

public class DataSourceContentField : IComputedIndexField
{
    public virtual string FieldName { get; set; }

    public virtual string ReturnType { get; set; }

    // TODO: Change to dictionary with supported template fields as the value
    internal static HashSet<ID> SupportedTemplates { get; set; } = new HashSet<ID>();

    public object ComputeFieldValue(IIndexable indexable)
    {
        var sitecoreIndexable = indexable as SitecoreIndexableItem;

        if (sitecoreIndexable == null) return null;

        // find renderings with datasources set
        var customDataSources = ExtractRenderingDataSourceItems(sitecoreIndexable.Item);

        // extract text from data sources
        var contentToAdd = customDataSources.SelectMany(GetItemContent).ToList();

        if (contentToAdd.Count == 0) return null;

        Log.Debug($"[{nameof(DataSourceContentField)}] Item: {sitecoreIndexable.AbsolutePath}, Content: {string.Join(" ", contentToAdd)}", this);

        return string.Join(" ", contentToAdd);
    }

    /// <summary>
    /// Finds all renderings on an item's layout details with valid custom data sources set and returns the data source items.
    /// </summary>
    protected virtual IEnumerable<Item> ExtractRenderingDataSourceItems(Item baseItem)
    {
        var currentLayoutXml = LayoutField.GetFieldValue(baseItem.Fields[FieldIDs.LayoutField]);
        if (string.IsNullOrEmpty(currentLayoutXml)) yield break;

        var layout = LayoutDefinition.Parse(currentLayoutXml);

        // loop over devices in the rendering
        for (var deviceIndex = layout.Devices.Count - 1; deviceIndex >= 0; deviceIndex--)
        {
            var device = layout.Devices[deviceIndex] as DeviceDefinition;

            if (device == null) continue;

            // loop over renderings within the device
            for (var renderingIndex = device.Renderings.Count - 1; renderingIndex >= 0; renderingIndex--)
            {
                var rendering = device.Renderings[renderingIndex] as RenderingDefinition;

                if (string.IsNullOrWhiteSpace(rendering?.Datasource)) continue;

                // if the rendering has a custom data source, we resolve the data source item and place its text fields into the content to add
                var dataSource = DataSourceHelper.ResolveDataSource(rendering.Datasource, baseItem);
                if (dataSource != baseItem && dataSource.IsDerivedFrom(SupportedTemplates))
                {
                    yield return dataSource;
                }
            }
        }
    }

    /// <summary>
    /// Extracts textual content from an item's fields
    /// </summary>
    protected virtual IEnumerable<string> GetItemContent(Item dataSource)
    {
        foreach (Field field in dataSource.Fields)
        {
            // this check is what Sitecore uses to determine if a field belongs in _content (see LuceneDocumentBuilder.AddField())
            if (!IndexOperationsHelper.IsTextField(new SitecoreItemDataField(field))) continue;

            var fieldValue = StringUtil.RemoveTags(field.Value ?? string.Empty);

            if (!string.IsNullOrWhiteSpace(fieldValue)) yield return fieldValue;
        }
    }
}
  • I would say this is a common approach, many projects I've worked on have had a "ParsedContent" field that represents all of the content that would actually be seen on a page. – Kasaku Oct 5 '16 at 8:53
1

The approach with a ComputedField has been done before (and often apparently) but we encountered some drawbacks - depending on the renderings in the project however. What we tried was to take the approach a step further and added an interface for each rendering that wants to add data to the index. Through that interface the rendering could send any data to the index for the page it was on.

Seemed very nice, but updates were an issue. We tried using the getDependencies to make sure we updated the "page" items when a datasource item was altered. This works, but it's not fully reliable as not all data is not always immediately in the datasource item - could be in the children of the datasource as well. In the end, we were stressing the index too much as it was updating everything the whole time.. didn't find a solution for this yet.

If your rendering architecture is a bit less complex, and all is linked very clearly and directly, it might work though.

  • I like the idea (as not all related data should be added to the page), but I can see the issues you encountered makes it too stressful for larger and more complex solutions. Again, I can't help thinking if it is just not doable and that a good old crawler would be better. – Anders Gjelstrup Oct 5 '16 at 9:05
  • Yes, the idea of having the rendering define what to index is nice, and possible. It's just those dependencies.. If you put the "pages" in a custom index you could also opt for a scheduled rebuild instead of constant updates. If the number of pages is not too high and the customer it might work. The results will be outdated between rebuilds, but that is also the case with crawlers... – Gatogordo Oct 5 '16 at 9:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.