We are using SOLR 5.4 and Sitecore 8.2 and we have implemented Full Content Search using _Content field which is displaying search result in Company Pages and various datasources linked to these company pages.
I need to display the Company Page in Search result, which is referring to the datasources item.
Now the question is, how can we write/configure that kind of computed index field?
From what I understand, you are looking for a way to get the datasources that your current page is referring to to be indexed along with said page. Several times now we have used the approach defined here:
https://kamsar.net/index.php/2014/05/indexing-subcontent/
Create a ComputedField Class
This chunk of code will trigger anytime an item is indexed and should push to your index whatever content is referred to as a datasource within your Renderings :
using System.Collections.Generic;
using System.Linq;
using Blade.Utility;
using Sitecore;
using Sitecore.ContentSearch;
using Sitecore.ContentSearch.ComputedFields;
using Sitecore.Data.Fields;
using Sitecore.Data.Items;
using Sitecore.Layouts;
namespace Foo.ContentSearch.ComputedFields
{
/// <summary>
/// Computed field that contains all textual content of items that are rendering data sources on the current item's layout details
/// </summary>
public class SubcontentField : IComputedIndexField
{
public object ComputeFieldValue(IIndexable indexable)
{
var sitecoreIndexable = indexable as SitecoreIndexableItem;
if (sitecoreIndexable == null) return null;
// find renderings with datasources set
var customDataSources = ExtractRenderingDataSourceItems(sitecoreIndexable.Item);
// extract text from data sources
var contentToAdd = customDataSources.SelectMany(GetItemContent).ToList();
if (contentToAdd.Count == 0) return null;
return string.Join(" ", contentToAdd);
}
/// <summary>
/// Finds all renderings on an item's layout details with valid custom data sources set and returns the data source items.
/// </summary>
protected virtual IEnumerable<Item> ExtractRenderingDataSourceItems(Item baseItem)
{
string currentLayoutXml = LayoutField.GetFieldValue(baseItem.Fields[FieldIDs.LayoutField]);
if (string.IsNullOrEmpty(currentLayoutXml)) yield break;
LayoutDefinition layout = LayoutDefinition.Parse(currentLayoutXml);
// loop over devices in the rendering
for (int deviceIndex = layout.Devices.Count - 1; deviceIndex >= 0; deviceIndex--)
{
var device = layout.Devices[deviceIndex] as DeviceDefinition;
if (device == null) continue;
// loop over renderings within the device
for (int renderingIndex = device.Renderings.Count - 1; renderingIndex >= 0; renderingIndex--)
{
var rendering = device.Renderings[renderingIndex] as RenderingDefinition;
if (rendering == null) continue;
// if the rendering has a custom data source, we resolve the data source item and place its text fields into the content to add
if (!string.IsNullOrWhiteSpace(rendering.Datasource))
{
// DataSourceHelper is a component of Blade
var dataSource = DataSourceHelper.ResolveDataSource(rendering.Datasource, baseItem);
if (dataSource != baseItem)
{
yield return dataSource;
}
}
}
}
}
/// <summary>
/// Extracts textual content from an item's fields
/// </summary>
protected virtual IEnumerable<string> GetItemContent(Item dataSource)
{
foreach (Field field in dataSource.Fields)
{
// this check is what Sitecore uses to determine if a field belongs in _content (see LuceneDocumentBuilder.AddField())
if (!IndexOperationsHelper.IsTextField(new SitecoreItemDataField(field))) continue;
string fieldValue = (field.Value ?? string.Empty).StripHtml();
if (!string.IsNullOrWhiteSpace(fieldValue)) yield return fieldValue;
}
}
public string FieldName { get; set; }
public string ReturnType { get; set; }
}
}
Update your configs
Add the reference to the ComputedFields class in your index config (please note the example below is referencing Lucene, you will likely have a similar structure with a different node indexConfiguration/defaultSolrIndexConfiguration would be my guess):
<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
<sitecore>
<contentSearch>
<indexConfigurations>
<defaultLuceneIndexConfiguration>
<fields hint="raw:AddComputedIndexField">
<!-- indexes subcontent contents into parent's _content field in the index (for better site search) -->
<field fieldName="_content" type="Foo.ContentSearch.ComputedFields.SubcontentField, Foo" />
</fields>
</defaultLuceneIndexConfiguration>
</indexConfigurations>
</contentSearch>
</sitecore>
</configuration>
A Step further
On top of this approach, often times we have a Site/Data/SearchSettings item located in a global place with regards to the site itself (and in a multi-site environment, we would have several of these items). On this item we have a multi-list field that would point at all renderings that would be considered "modules" or "components" to a Page item. From there you can pick and choose which datasource/subcontent items get indexed and which don't (some items may be data centric and some may be html content). It's a pretty small addition (a settings item and some code in the computed field class) but it helps give content editors granularity into what is indexed:
ComputedField(IIndexable indexable) method:
...
var sitecoreIndexable = indexable as SitecoreIndexableItem;
if (sitecoreIndexable == null) return null;
var sc = new SitecoreContext(Sitecore.Context.ContentDatabase);
var searchSettings = sc.GetItem<Search_Settings>(searchSettingsItem.ID.ToString().ToUpper());
if (searchSettings == null)
return null;
indexedRenderings = searchSettings.Indexed_Renderings.Select(i => new ID(i).ToString()).ToList();
var customDataSources = ExtractRenderingDataSourceItems(item, indexedRenderings);
...
ExtractRendringDatasourceItems(Item baseItem, List<string> indexedRenderings) method (added parameter):
...
if (rendering == null) continue;
// only index valid renderings
if (!indexedRenderings.Contains(rendering.RenderingID.ToString().ToUpper())) continue;
// if the rendering has a custom data source, we resolve the data source item and place its text fields into the content to add
if (!string.IsNullOrWhiteSpace(rendering.Datasource))
...
Dependencies
Lastly, since your datasources will likely change, it won't necessarily get the change in the index since the parent doesn't change. You will want to make the subcontent/datasource change trigger a bubble-up index (http://www.techphoria414.com/Blog/2013/November/Sitecore-7-Computed-Fields-All-Templates-and-Datasource-Content).
Create GetDatasourceDependencies pipeline
This will help gather the referrers and flag them as dependencies:
public class GetDatasourceDependencies : BaseProcessor
{
public override void Process(GetDependenciesArgs context)
{
Func<ItemUri, bool> func = null;
Assert.IsNotNull(context.IndexedItem, "indexed item");
Assert.IsNotNull(context.Dependencies, "dependencies");
Item item = (Item)(context.IndexedItem as SitecoreIndexableItem);
if (item != null)
{
if (func == null)
{
func = uri => (bool)((uri != null) && ((bool)(uri != item.Uri)));
}
System.Collections.Generic.IEnumerable<ItemUri> source = Enumerable.Where<ItemUri>(from l in Globals.LinkDatabase.GetReferrers(item, FieldIDs.LayoutField) select l.GetSourceItem().Uri, func).Distinct<ItemUri>();
context.Dependencies.AddRange(source.Select(x => (SitecoreItemUniqueId)x));
}
}
}
Add the pipeline to your config:
<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
<sitecore>
<pipelines>
<indexing.getDependencies help="Processors should derive from Sitecore.ContentSearch.Pipelines.GetDependencies.BaseProcessor">
<!-- When indexing an item, make sure any items for whom it is a datasource get re-indexed as well -->
<processor type="Foo.ContentSearch.GetDatasourceDependencies, Foo"/>
</indexing.getDependencies>
</pipelines>
</sitecore>
</configuration>
Hopefully this helps!
Similar to the answer provided by @vandsh, but with less dependencies I created this gist, which is outlined below.
Note: This method assumes all datasources are referenced by ID. It relies on the links database to find linked items, which would fail if relying on a query.
This avoids indexing renderings such as Navigation or Related Articles, which really aren't pertinent to the actual page content.
You'll want to update this list as you build your module library with relevant text-heavy renderings.
<configuration>
<sitecore>
<indexedRenderings>
<indexedRendering name="RichText" id="{4F8942FF-C4E4-408A-B3D5-4657EE5050CA}" />
<indexedRendering name="EmployeeBio" id="{53F8166B-6627-4C1B-BED5-FBD0B36EC527}" />
<indexedRendering name="HighlightStatic" id="{75100A87-B7B0-4D1F-BE85-1A8FC6342514}" />
<indexedRendering name="IntroText" id="{89D7EE42-A5E1-4F24-B247-BFD382E501D4}" />
</indexedRenderings>
</sitecore>
</configuration>
using System.Collections.Generic;
using System.Linq;
using System.Xml;
using Sitecore.Configuration;
using Sitecore.Data;
using Sitecore.Xml;
namespace Common.Sitecore.ConfigElements
{
public class IndexedRenderings
{
public string Name { get; set; }
public ID Id { get; set; }
public static List<IndexedRenderings> IndexableRenderings()
{
return (from XmlNode node in Factory.GetConfigNodes("indexedRenderings/indexedRendering")
select new IndexedRenderings
{
Name = XmlUtil.GetAttribute("name", node),
Id = new ID(XmlUtil.GetAttribute("id", node))
}).ToList();
}
}
}
This field will contain the filtered text of all datasources (allowed by config section) added to the page item. The contents is stored in the index as renderedContent. It checks both the Layout and FinalLayout of the item for renderings that are marked as allowed, then iterates on the fields of the datasource, ultimately piping everything to the same renderedContent string.
Note: You probably don't want to use this field for display on a search results page, for example, but can still be leveraged to provide relevant results.
<configuration>
<sitecore>
<contentSearch>
<indexConfigurations>
<defaultLuceneIndexConfiguration> <!-- swap for your configuration, e.g. SOLR!!! -->
<documentOptions>
<fields hint="raw:AddComputedIndexField">
<field fieldName="renderedContent" storageType="no" indexType="tokenized">Common.Sitecore.Search.ComputedFields.RenderedContent,Common</field>
</fields>
</documentOptions>
</defaultLuceneIndexConfiguration>
</indexConfigurations>
</contentSearch>
</sitecore>
</configuration>
using System.Collections.Generic;
using System.Linq;
using System.Text.RegularExpressions;
using Sitecore;
using Sitecore.ContentSearch;
using Sitecore.ContentSearch.ComputedFields;
using Sitecore.Data;
using Sitecore.Data.Fields;
using Sitecore.Data.Items;
using Sitecore.WordOCX.HtmlDocument;
using Common.Sitecore.ConfigElements;
namespace Common.Sitecore.Search.ComputedFields
{
public class RenderedContent : AbstractComputedIndexField
{
private IEnumerable<ID> _indexableRenderings;
public override object ComputeFieldValue(IIndexable indexable)
{
var item = (Item)(indexable as SitecoreIndexableItem);
if (item == null)
return null;
_indexableRenderings = IndexedRenderings.IndexableRenderings().Select(i => i.Id);
if (!_indexableRenderings.Any())
return null;
var deviceItem = item.Database.Resources.Devices.GetAll().FirstOrDefault(d => d.IsDefault);
if (deviceItem == null)
return null;
var itemDatasources = ExtractRenderingDataSourceItems(item, _indexableRenderings, item.Fields[FieldIDs.FinalLayoutField], deviceItem).ToList(); // for later `Distinct()`
itemDatasources.AddRange(ExtractRenderingDataSourceItems(item, _indexableRenderings, item.Fields[FieldIDs.LayoutField], deviceItem));
itemDatasources = itemDatasources.Distinct().ToList();
if (!itemDatasources.Any())
return null;
var content = new List<string>();
foreach (var itemDatasource in itemDatasources)
GetItemContent(itemDatasource, content);
if (!content.Any())
return null;
var aggregateContent = content.Aggregate((a, b) => $"{a} {b}");
aggregateContent = StripLineBreaksAndWhiteSpace(aggregateContent);
return aggregateContent;
}
private static IEnumerable<Item> ExtractRenderingDataSourceItems(Item item, IEnumerable<ID> indexableRenderings, Field layoutField, DeviceItem deviceItem)
{
if (layoutField == null)
return Enumerable.Empty<Item>();
var layoutXml = LayoutField.GetFieldValue(layoutField);
if (string.IsNullOrEmpty(layoutXml))
return Enumerable.Empty<Item>();
var renderings = item.Visualization.GetRenderings(deviceItem, false);
if (renderings == null || renderings.Length == 0)
return Enumerable.Empty<Item>();
var datasourceItems = new List<Item>();
for (var renderingIndex = renderings.Length - 1; renderingIndex >= 0; renderingIndex--)
{
var rendering = renderings[renderingIndex];
if (rendering == null || rendering.Database != item.Database)
continue;
if (!indexableRenderings.Contains(rendering.RenderingID))
continue;
var datasourceId = rendering.Settings.DataSource;
if (!ID.IsID(datasourceId))
continue;
var datasourceItem = item.Database.GetItem(new ID(datasourceId));
if(datasourceItem == null)
continue;
datasourceItems.Add(datasourceItem);
}
return datasourceItems;
}
/// <summary>
/// Extracts textual content from an item's fields
/// </summary>
private static void GetItemContent(Item dataSource, ICollection<string> content)
{
//.Where(field => !field.Name.StartsWith("__")
foreach (Field field in dataSource.Fields)
{
if (!IndexOperationsHelper.IsTextField(new SitecoreItemDataField(field)))
continue;
var fieldValue = StripHtmlTags(field.Value ?? string.Empty);
if (!string.IsNullOrWhiteSpace(fieldValue))
content.Add(fieldValue);
}
foreach (Item child in dataSource.Children)
{
GetItemContent(child, content);
}
}
private static string StripHtmlTags(string source)
{
if (source == null)
return null;
var doc = new HtmlDocument();
doc.LoadHtml(source);
return doc.DocumentNode.InnerText;
}
private static string StripLineBreaksAndWhiteSpace(string source)
{
if(string.IsNullOrWhiteSpace(source))
return null;
source = source.Replace(System.Environment.NewLine, "");
source = Regex.Replace(source, @"\s+", " ");
return source.Trim(' ');
}
}
}
Since a content author will (not might) update a datasource and not update the associated page, the below changes instruct Sitecore to find any pages that may be relying on the datasource and to reindex them. If relying on a shared datasource, all pages relying on that datasource will update.
<configuration>
<sitecore>
<pipelines>
<indexing.getDependencies>
<processor type="Common.Sitecore.Search.GetDependencies.GetDatasourceDependencies, Common"/>
</indexing.getDependencies>
</pipelines>
</sitecore>
</configuration>
using System.Linq;
using Sitecore;
using Sitecore.ContentSearch;
using Sitecore.ContentSearch.Pipelines.GetDependencies;
using Sitecore.Data;
using Sitecore.Data.Items;
using Sitecore.Diagnostics;
namespace Common.Sitecore.Search.GetDependencies
{
public class GetDatasourceDependencies : BaseProcessor
{
public override void Process(GetDependenciesArgs context)
{
Assert.IsNotNull(context.IndexedItem, "indexed item");
Assert.IsNotNull(context.Dependencies, "dependencies");
var item = (Item)(context.IndexedItem as SitecoreIndexableItem);
if (item == null)
return;
bool Func(ItemUri uri) => (uri != null) && uri != item.Uri;
var source = (
from l
in Globals.LinkDatabase.GetReferrers(item, FieldIDs.LayoutField)
select l.GetSourceItem().Uri)
.Where(Func)
.ToList();
source.AddRange(
(
from l
in Globals.LinkDatabase.GetReferrers(item, FieldIDs.FinalLayoutField)
select l.GetSourceItem().Uri)
.Where(Func)
);
source = source.Distinct().ToList();
context.Dependencies.AddRange(source.Select(x => (SitecoreItemUniqueId)x));
}
}
}
At this point you can write queries against this field as you typically would. Example of a custom search object with the field implemented:
using System.Runtime.Serialization;
using Sitecore.ContentSearch;
using Sitecore.ContentSearch.SearchTypes;
namespace Common.Sitecore.Seach
{
public class CustomSearchResultItem : SearchResultItem
{
[DataMember]
[IndexField("renderedContent")]
public string RenderedContent { get; set; }
}
}
Class to read config all together.