How can I show a list of related/similar pages?

Question

Our website needs to show a list of related questions, very much in the same way as it works on StackExchange:

Is there some sort of built-in functionality to achieve this? What about existing Marketplace components?

Answer 1

What are the relations?

As we briefly discussed in comments; how articles relate can be a very complicated setup. In this instance the assumption will be that of category affinity. So put very simply; the more categories two articles have in common, the more related they are.

While a very simplified approach, this is actually a surprisingly good starting point for many scenarios. If the need is, to be more advanced, it would be possible to differentiate the weighting based on super- and sub-categories. For this example, I'll not go into that.

The setup

Much as per my 2009 blog post listed above (although I am even more lazy with the test data this time around) - I've done a simple Sitecore IA to demonstrate this.

Template setup

A very simple Article and Category template. Article holds a MultilistField to tag categories. Can be any of Sitecores Multi link field types, I use TreelistEx here.

Content Setup

Just a few articles.

A bit of test code to demonstrate

A quick .aspx mockup. Nothing special here.

        var articlesRoot = Sitecore.Context.Database.GetItem("/sitecore/content/Home/Articles");
        var articles = articlesRoot.GetChildren();

        foreach (Item article in articles)
        {
            Response.Write("<strong>Article: " + article.Paths.FullPath + "</strong><br />");
            foreach (var relatedArticle in GetRelatedArticles(article))
            {
                // Do something with these
                Response.Write(relatedArticle.Paths.FullPath + "<br />");
            }
            Response.Write("<hr />");
        }

The core of the problem

With all of that out of the way; I will demonstrate how to solve this using the built-in Sitecore LinkDatabase. A little known gem, even if I do my best to promote it whenever I can ;-)

In short steps; this is how the method GetRelatedArticles operates.

• Get all referenced categories from the article in question
• For each of those categories, collect all articles that use the category
• Record the number of articles count for the category

In code, this comes out as:

    private IEnumerable<Item> GetRelatedArticles(Item article)
    {
        // Dirty; no argument checking, no null checking. Just barebones.

        var referenceCount = new Dictionary<ID,int>();
        MultilistField mlf = article.Fields["Article Categories"];

        foreach (Item categoryItem in mlf.GetItems())
        {
            var articleLinksInCategory = Globals.LinkDatabase.
                GetItemReferrers(categoryItem, false).
                Where(al => al.SourceFieldID == mlf.InnerField.ID);

            foreach (var articleLink in articleLinksInCategory)
            {
                if (articleLink.SourceItemID != article.ID)
                {
                    if (referenceCount.ContainsKey(articleLink.SourceItemID))
                        referenceCount[articleLink.SourceItemID]++;
                    else
                        referenceCount[articleLink.SourceItemID] = 1;
                }
            }
        }

The code then continues to:

• Build up a list of all the referenced articles, along with their count

Comes out like this:

        var articlesWithCount = new List<Tuple<ID,int>>();

        foreach (var articleReference in referenceCount.Keys)
        {
            articlesWithCount.Add(
                new Tuple<ID, int>(articleReference, referenceCount[articleReference])
            );
        }

• Sort the articles, so the most relevant comes out top

Like this:

        articlesWithCount.Sort((x, y) => y.Item2.CompareTo(x.Item2));

And finally, return just the Item part of the Tuple List.

        return articlesWithCount.ConvertAll(i => article.Database.GetItem(i.Item1));
    }

For the copy/pastyness, here is the method in its completeness:

    private IEnumerable<Item> GetRelatedArticles(Item article)
    {
        // Dirty; no argument checking, no null checking. Just barebones.

        var referenceCount = new Dictionary<ID,int>();
        MultilistField mlf = article.Fields["Article Categories"];

        foreach (Item categoryItem in mlf.GetItems())
        {
            var articleLinksInCategory = Globals.LinkDatabase.
                GetItemReferrers(categoryItem, false).
                Where(al => al.SourceFieldID == mlf.InnerField.ID);

            foreach (var articleLink in articleLinksInCategory)
            {
                if (articleLink.SourceItemID != article.ID)
                {
                    if (referenceCount.ContainsKey(articleLink.SourceItemID))
                        referenceCount[articleLink.SourceItemID]++;
                    else
                        referenceCount[articleLink.SourceItemID] = 1;
                }
            }
        }

        var articlesWithCount = new List<Tuple<ID,int>>();

        foreach (var articleReference in referenceCount.Keys)
        {
            articlesWithCount.Add(
                new Tuple<ID, int>(articleReference, referenceCount[articleReference])
            );
        }

        articlesWithCount.Sort((x, y) => y.Item2.CompareTo(x.Item2));

        return articlesWithCount.ConvertAll(i => article.Database.GetItem(i.Item1));
    }

And if you only need 10 article references, a bit of late night lazy LINQ solves that too. The less items you resolve in this method, the better it performs.

return articlesWithCount.Take(10).ToList().ConvertAll(i => article.Database.GetItem(i.Item1));

Performance?

It really is very good. While I did no measurements in this specific test, I have previously tested LinkDatabase performance and it continues to come up very high.

If you reach the breaking point of this; the next step is to implement this via Sitecore ContentSearch. Has a larger setup footprint but will scale you into the 100s of 1000s of articles in as many categories.

Answer 2

Some have already mentioned about the use of TAGS and I would like to reinforce this is a very good practice. It is considerably an easy solution to develop and the content editors will appreciate the simplicity.

Here is a suggestion of how you could implement this:

Create the following structure (use a template and name it 'Questions Listing' for example).

Create a "Question" template containing a 'Treelist' field called 'TAGS'. This filed will be used to filter your questions.

You see, the 'Questions Listing' will only list 'Question' items!

Now on each 'question' you can set the related tags.

I hope this is useful for you.

Regards

Answer 3

Sitecore doesn't directly provide this feature - rather it provides a tool-box of APIs to help you build the feature to suit your needs. Off the top of my head I'm not aware of a Marketplace module which solves this particular issue - but I've not searched extensively, so I may be out of date there.

If you're assembling it yourself, related content needs two things: It needs data to describe relations and it needs some sort of query mechanism to read this data at runtime.

Off the top of my head, I can think of a few common patterns for describing the relationship between pages:

1. Direct page links
   A TreeList type field on your page's Template can allow an editor to directly select other pages that they want to be "related". This works best for smaller sets of content, or for scenarios where you want very direct control of the relationship.
   When you process this data all you have to do is load the items that the field points to, and present that list to the user.

2. Tagging
   You can define some items in your content tree to represent a tag taxonomy. A MultiList or TreeList field on your pages can then allow editors to pick a set of tags for each page. This tends to work better when you want pages to describe a category they belong to, rather than a direct relationship with other pages. (Which is more like the example you cite above)
   To render related items from tag data you need to take the set of tag IDs recorded on the current page and then find "all items which include these tags, but not the context item". Whether you choose to show items which include "any of these tags" or "all of these tags" depends on your business rules.

3. Automated tags
   It's also possible to generate "tag" style data automatically if you have some rules to implement. You might match keywords in text fields, or compute something from other metadata for example. This could be done as an ordinary field that is populated by a Save event, or it could be done as a Computed Field for the search index. (There was also a very interesting blog post recently showing you might even do this via 3rd party Cognitive Services APIs)
   Processing this data depends entirely on what you choose to record as a result of your generation algorithm. I'd probably aim for recording the IDs of the pages you deem to be related, as that makes presentation easiest.

Once you've defined a relationship, you then need to be able to query that data to render the UI. Again, there are assorted options here:

1. As Mark suggests in the comments above, Sitecore has a built-in database of relationships between items called the Link Database. This has API calls which can find you the items which the context item has a relationship to. This will include things like templates as well as your related pages or tags - so you do need to be careful to filter this data correctly.

2. If you configure the ContentSearch framework to index the data in your relationship fields, you can also use search queries to find the target items and return data. If your index stores the appropriate data then this approach can be used to render the UI without the need to access the Sitecore Database directly, which may offer performance benefits.

3. You can also use the standard Sitecore APIs to load the items based on the IDs stored in your fields.

Which approach to processing the data you choose is probably down to which fits best with the site you're building. As your content gets bigger and more complex, the ContentSearch approach is likely to perform better than the others - but it will probably require more code and configuration to make it work.