Definition of content on a page

Question

I have thought about how to best assemble a "page" document in Lucene/Solr to later use in Sitecore when searching for most relevant pages to show to a user.

A page consist of many elements which is typically indexed separately in Sitecore.

How would I best assemble the item that represents a given "page"? As I see it I would need to extract all reference from the actual page item, look up all related items and store their content in the document representing the "page" document.

This seems kind of bulky if that needs to be performed each time a page item is changed.

Seems easier just to crawl all pages the old fashion way.

What is the best solution for indexing whole pages?

Answer 1

One approach that we have used is to index the content of all data sources in a single computed field. You can search that field when searching the index. This can probably be improved, but it's a start.

I believe this is also the approach the are using in the Habitat project.

Copied/inspired from: http://kamsar.net/index.php/2014/05/indexing-subcontent/

public class DataSourceContentField : IComputedIndexField
{
    public virtual string FieldName { get; set; }

    public virtual string ReturnType { get; set; }

    // TODO: Change to dictionary with supported template fields as the value
    internal static HashSet<ID> SupportedTemplates { get; set; } = new HashSet<ID>();

    public object ComputeFieldValue(IIndexable indexable)
    {
        var sitecoreIndexable = indexable as SitecoreIndexableItem;

        if (sitecoreIndexable == null) return null;

        // find renderings with datasources set
        var customDataSources = ExtractRenderingDataSourceItems(sitecoreIndexable.Item);

        // extract text from data sources
        var contentToAdd = customDataSources.SelectMany(GetItemContent).ToList();

        if (contentToAdd.Count == 0) return null;

        Log.Debug($"[{nameof(DataSourceContentField)}] Item: {sitecoreIndexable.AbsolutePath}, Content: {string.Join(" ", contentToAdd)}", this);

        return string.Join(" ", contentToAdd);
    }

    /// <summary>
    /// Finds all renderings on an item's layout details with valid custom data sources set and returns the data source items.
    /// </summary>
    protected virtual IEnumerable<Item> ExtractRenderingDataSourceItems(Item baseItem)
    {
        var currentLayoutXml = LayoutField.GetFieldValue(baseItem.Fields[FieldIDs.LayoutField]);
        if (string.IsNullOrEmpty(currentLayoutXml)) yield break;

        var layout = LayoutDefinition.Parse(currentLayoutXml);

        // loop over devices in the rendering
        for (var deviceIndex = layout.Devices.Count - 1; deviceIndex >= 0; deviceIndex--)
        {
            var device = layout.Devices[deviceIndex] as DeviceDefinition;

            if (device == null) continue;

            // loop over renderings within the device
            for (var renderingIndex = device.Renderings.Count - 1; renderingIndex >= 0; renderingIndex--)
            {
                var rendering = device.Renderings[renderingIndex] as RenderingDefinition;

                if (string.IsNullOrWhiteSpace(rendering?.Datasource)) continue;

                // if the rendering has a custom data source, we resolve the data source item and place its text fields into the content to add
                var dataSource = DataSourceHelper.ResolveDataSource(rendering.Datasource, baseItem);
                if (dataSource != baseItem && dataSource.IsDerivedFrom(SupportedTemplates))
                {
                    yield return dataSource;
                }
            }
        }
    }

    /// <summary>
    /// Extracts textual content from an item's fields
    /// </summary>
    protected virtual IEnumerable<string> GetItemContent(Item dataSource)
    {
        foreach (Field field in dataSource.Fields)
        {
            // this check is what Sitecore uses to determine if a field belongs in _content (see LuceneDocumentBuilder.AddField())
            if (!IndexOperationsHelper.IsTextField(new SitecoreItemDataField(field))) continue;

            var fieldValue = StringUtil.RemoveTags(field.Value ?? string.Empty);

            if (!string.IsNullOrWhiteSpace(fieldValue)) yield return fieldValue;
        }
    }
}

Answer 2

The approach with a ComputedField has been done before (and often apparently) but we encountered some drawbacks - depending on the renderings in the project however. What we tried was to take the approach a step further and added an interface for each rendering that wants to add data to the index. Through that interface the rendering could send any data to the index for the page it was on.

Seemed very nice, but updates were an issue. We tried using the getDependencies to make sure we updated the "page" items when a datasource item was altered. This works, but it's not fully reliable as not all data is not always immediately in the datasource item - could be in the children of the datasource as well. In the end, we were stressing the index too much as it was updating everything the whole time.. didn't find a solution for this yet.

If your rendering architecture is a bit less complex, and all is linked very clearly and directly, it might work though.