Understanding Search Based on _content field

Question

We have a client site , with SOLR for searching products . The Code base for returning the search results is shown below. The products searched are based on search term entered by user

Code

var query = context.GetQueryable<SearchResultItem>();
                    query = query.AddContentSearchQuery(this._searchTerm);
                    query = query.Filter(predicate);

AddContentSearchQuery Matches the search text

public static IQueryable<SearchResultItem> AddContentSearchQuery(this IQueryable<SearchResultItem> query, string searchTerms)
        {
            if (string.IsNullOrEmpty(searchTerms))
                return query;
            searchTerms = searchTerms.Trim(' ').Replace(' ', '@');

            return query.Where(i => i.Content == searchTerms);
        }

here i.Content is from Sitecore.ContentSearch.SearchTypes.SearchResultItem

[DataMember]
**[IndexField("_content")]**
public virtual string Content { get; set; }

Can you please help me in understanding the analyzer used here ( Does it tokenize the search content)

This is the entry from Sitecore.ContentSearch.Solr.DefaultIndexConfiguration.config

<field fieldName="_content"                       returnType="string"     type="Sitecore.ContentSearch.ComputedFields.MediaItemContentExtractor,Sitecore.ContentSearch">
              <mediaIndexing ref="contentSearch/indexConfigurations/defaultSolrIndexConfiguration/mediaIndexing"/>
            </field>

What I understand is the _content field indexes the full content of the website. but is it also Tokenized ?

Answer 1

If you look at the _content field definition in the Solr schema.xml file you see this:

<field name="_content" type="text_general" indexed="true" stored="false" />

The field is indexed and has a type of text_general

Here is the definition for the text_general field type:

<fieldType name="text_general" class="solr.TextField" positionIncrementGap="100">
  <analyzer type="index">
    <tokenizer class="solr.StandardTokenizerFactory" />
    <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" />
    <!-- in this example, we will only use synonyms at query time
    <filter class="solr.SynonymFilterFactory" synonyms="index_synonyms.txt" ignoreCase="true" expand="false"/>
    -->
    <filter class="solr.LowerCaseFilterFactory" />
  </analyzer>
  <analyzer type="query">
    <tokenizer class="solr.StandardTokenizerFactory" />
    <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" />
    <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true" />
    <filter class="solr.LowerCaseFilterFactory" />
  </analyzer>
</fieldType>

So it is tokenized using the solr.StandardTokenizerFactory

Hope that helps

Answer 2

If you're searching against 'products' you would perhaps be better off search against specific fields rather than the generic 'content' field?

I realise I'm answering a question that hasn't explicitly been asked, but it may be helpful anyway.

If searching multiple words in an 'AND' fashion that allows the user to reduce the number of results by adding words, you may want to filter out 'stop words', otherwise you will always receive zero results if the search contains one because your query says it must be present and it is ignored by the tokenizer even when present. You mentioned you're using SOLR but for Lucene these are found at Lucene.Net.Analysis.Standard.StandardAnalyzer.STOP_WORDS_SET. Stop words would be removed form your terms before querying.

The sample below does this and also uses LIKE instead of equals to allow for minor typos and additionally boosts results where the term is the same as the item name.

var terms = searchTerms.Split(new[] { ' ' }, StringSplitOptions.RemoveEmptyEntries).ToList();
terms = terms.Except(Lucene.Net.Analysis.Standard.StandardAnalyzer.STOP_WORDS_SET, StringComparer.OrdinalIgnoreCase).ToList();

var termsPredicate = PredicateBuilder.True<SearchResultItem>();

foreach (var term in terms)
{
    // AND = must contain every word
    termsPredicate = termsPredicate.And(  
    PredicateBuilder.False<_Base_Article>() // OR = in either of these two fields
       .Or(p => p.Name.Like(word, 0.8f).Boost(2.0f)) // minimal likeness 0.8, boost direct name match
       .Or(p => p.Content.Like(word, 0.8f))
    );
}