6

I have the following code to return a List of Items in a Lucene index:

public List<Item> GetSearchResults(ISearchIndex index, string searchTerm = "")
{
    var results = new List<Item>();

    using (var searchContext = index.CreateSearchContext())
    {
        if (string.IsNullOrWhiteSpace(searchTerm))
        {
            results = searchContext.GetQueryable<SearchResultItem>().Select(n => n.GetItem()).ToList();
        }
        else
        {
            searchTerm = searchTerm.Trim().ToLower();
            results = searchContext.GetQueryable<SearchResultItem>()
                .Where(n => n.Content.ToLower().Contains(searchTerm) 
                    || n.Url.ToLower().Contains(searchTerm) 
                    || n.Name.ToLower().Contains(searchTerm) 
                    || n.Fields["tags"].ToString().ToLower().Contains(searchTerm))
                .Select(n => n.GetItem())
                .ToList();
        }
    }

    return results;
}

If searchTerm is not null or empty I would like to sort by a weighting based on the number of occurrences of the search term. Is this at all possible?

7

Lucene will automatically apply that kind of weighting. It will give each record a "score" based on how well the document matches the term. A lot will be dependent on how you have crafted your search query of course.

You can influence the score by boosting. This is the how that affects the query:

Lucene allows influencing search results by "boosting" in more than one level:

  • Document level boosting - while indexing - by calling document.setBoost() before a document is added to the index.
  • Document's Field level boosting - while indexing - by calling field.setBoost() before adding a field to the document (and before adding the document to the index).
  • Query level boosting - during search, by setting a boost on a query clause, calling Query.setBoost().

Indexing time boosts are preprocessed for storage efficiency and written to the directory (when writing the document) in a single byte (!) as follows: For each field of a document, all boosts of that field (i.e. all boosts under the same field name in that doc) are multiplied. The result is multiplied by the boost of the document, and also multiplied by a "field length norm" value that represents the length of that field in that doc (so shorter fields are automatically boosted up). The result is decoded as a single byte (with some precision loss of course) and stored in the directory. The similarity object in effect at indexing computes the length-norm of the field.

This composition of 1-byte representation of norms (that is, indexing time multiplication of field boosts & doc boost & field-length-norm) is nicely described in Fieldable.setBoost().

Encoding and decoding of the resulted float norm in a single byte are done by the static methods of the class Similarity: encodeNorm() and decodeNorm(). Due to loss of precision, it is not guaranteed that decode(encode(x)) = x, e.g. decode(encode(0.89)) = 0.75. At scoring (search) time, this norm is brought into the score of document as norm(t, d), as shown by the formula in Similarity.

Source

Out of the box, Sitecore doesn't apply an index time boosting to your custom fields, so it will just apply a score based on how relevant your searchTerm is to the field. If you wanted to see the score, you can use the .GetResult method.

public List<Item> GetSearchResults(ISearchIndex index, string searchTerm = "")
{
    var results = new List<Item>();

    using (var searchContext = index.CreateSearchContext())
    {
        if (string.IsNullOrWhiteSpace(searchTerm))
        {
            results = searchContext.GetQueryable<SearchResultItem>().Select(n => n.GetItem()).ToList();
        }
        else
        {
            searchTerm = searchTerm.Trim().ToLower();
            var query = searchContext.GetQueryable<SearchResultItem>()
                .Where(n => n.Content.ToLower().Contains(searchTerm) 
                    || n.Url.ToLower().Contains(searchTerm) 
                    || n.Name.ToLower().Contains(searchTerm) 
                    || n.Fields["tags"].ToString().ToLower().Contains(searchTerm));


            var resultsWithScore = query.GetResult();

            var firstObject = resultsWithScore.FirstOrDefault();
            var score = firstObject.Score;
            var firstItem = firstObject.Document;

            results = resultsWithScore.Hits.Select(n => n.Document.GetItem()).ToList();
        }
    }

    return results;
}

If you wanted to make Content have a higher importance than Url in the search, then you can boost by field at query time:

            var query = searchContext.GetQueryable<SearchResultItem>()
                .Where(n => n.Content.ToLower().Contains(searchTerm).Boost(2f) 
                    || n.Url.ToLower().Contains(searchTerm) 
                    || n.Name.ToLower().Contains(searchTerm) 
                    || n.Fields["tags"].ToString().ToLower().Contains(searchTerm));

You can use all these features to tune your query. For the basic requirement you outlined above, Lucene should handle that without any extra coding.

|improve this answer|||||
  • Thank you! Is it also possible to index the contents of a linked PDF -- referenced in n.Fields["file"]? – eat-sleep-code Nov 1 '16 at 17:50
  • 2
    The content of a pdf will be indexed as a separate item (the media item) - it is not included in the item where it is referenced. – Gatogordo Nov 1 '16 at 17:54
  • So, no way to boost the above search results based on the content of the PDF? – eat-sleep-code Nov 1 '16 at 17:55
5

This should be done automatically for you. As you are not applying an OrderBy, the results returned will be in the order according to their "Score", which is the value Lucene assigns to each search result to determine how relevant a result is - the higher the better.

I would expect this to already be working in your query, but you may need to call GetResults() rather than your Select and then get the items via the Hits property inside. I do think it should work without this though.

Scoring background

In a Lucene search, multiple occurrences of a search term will cause that result to score higher. However, there are other factors that go into Lucene scoring. For example, the shorter a field is with a matching searchTerm, the higher the score - i.e. "searchTerm" will actually score higher than "searchTerm and also searchTerm but most of all searchTerm.". The idea here is that the former is deemed more relevant as it is wholly about the searchTerm.

I strongly recommend when working with search that you take a look at the queries that get logged out into the data/logs/search.log files. Using a tool such as Luke you can load in the Lucene indexes and manually execute the queries to see the results. Luke will give you an explanation for how each score was calculated for each search result.

You should be able to setup an index field that ignores the field-length when indexing data - which is referred to as normalization data. However, looking at the Sitecore code, I think they may have blocked this method off in their field configuration syntax, but I'm not sure.

|improve this answer|||||
3

Sitecore will indeed sort the results by default on a "Score", based on the relevance of the hit and calculated by the underlying index - Lucene in your case.

I do think though that you will need to call the GetResults() to get that functionality. So change .Select(n => n.GetItem()) to GetResults(). This will give you SearchResults - return the items based on the hit.Document in every hit from results.Hits.

Your resulting "hit" objects will have a Score property (float) containing the score value. You could use this property also if you want to use custom sorting and still include the score.

Note that if you would want to use boosting, that would not work as you are using a "contains" and they do not work together.

|improve this answer|||||

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.