How to apply stop words at Solr query time?

Question

I'm having some fun with Solr and here is my current issue.

I've applied some stop words using the stopwords.txt file and rebuilt my indexes. Everything looks fine here, I can see that those stop words are no longer indexed:

All that being said Query Time is my issue.

Firstly, here is my text_general field type config on managed-schema:

<fieldType name="text_general" class="solr.TextField" positionIncrementGap="100" multiValued="false">
    <analyzer type="index">
      <tokenizer class="solr.StandardTokenizerFactory"/>
      <filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="true"/>
      <filter class="solr.LowerCaseFilterFactory"/>
    </analyzer>
    <analyzer type="query">
      <tokenizer class="solr.StandardTokenizerFactory"/>
      <filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="true"/>
      <filter class="solr.SynonymFilterFactory" expand="true" ignoreCase="true" synonyms="synonyms.txt"/>
      <filter class="solr.LowerCaseFilterFactory"/>
    </analyzer>
</fieldType>

Since the StopFilterFactory is there on both Index and Query time I'm wondering why queries with stop words are keeping those words during execution.

Let's say I'm quering Designed for you.

Since for is a stopword my query should really be Designed you.

Here is the code handling the query:

var keywordsFilter = PredicateBuilder.True<UnifiedSearchResultItemWithLocalDatasource>();
            foreach (var keyword in keywords)
            {
                var keywordFilter = PredicateBuilder.False<UnifiedSearchResultItemWithLocalDatasource>();
                keywordFilter = keywordFilter.Or(item => item.UnifiedTitle.Like(keyword, 0.8f).Boost(8f))
                    .Or(item => item.UnifiedTitle.StartsWith(keyword).Boost(8f))
                    .Or(item => item.UnifiedSiteSection.Like(keyword, 0.8f).Boost(4f))
                    .Or(item => item.UnifiedSiteSection.StartsWith(keyword).Boost(4f))
                    .Or(item => item.UnifiedDescription.Like(keyword, 0.8f).Boost(2f))
                    .Or(item => item.UnifiedDescription.StartsWith(keyword).Boost(2f));

                keywordsFilter = keywordsFilter.And(keywordFilter);
            }

And here is the Search log with the generated Solr query:

16776 17:32:29 INFO Solr Query - ?q=((-unifiedtitle_t:("") unifiedtitle_t:[* TO ]) AND ((((unifiedtitle_t:(designed~0.8))^8 OR (unifiedtitle_t:(designed))^8 OR (unifiedsitesection_t:(designed~0.8))^4 OR (unifiedsitesection_t:(designed*))^4 OR (unifieddescription_t:(designed~0.8))^2 OR (unifieddescription_t:(designed*))^2) AND ((unifiedtitle_t:(for~0.8))^8 OR (unifiedtitle_t:(for*))^8 OR (unifiedsitesection_t:(for~0.8))^4 OR (unifiedsitesection_t:(for*))^4 OR (unifieddescription_t:(for~0.8))^2 OR (unifieddescription_t:(for*))^2)) AND ((unifiedtitle_t:(you~0.8))^8 OR (unifiedtitle_t:(you*))^8 OR (unifiedsitesection_t:(you~0.8))^4 OR (unifiedsitesection_t:(you*))^4 OR (unifieddescription_t:(you~0.8))^2 OR (unifieddescription_t:(you*))^2)))&start=0&rows=10&fl=*,score&fq=((_latestversion:(True) AND has_presentation_b:(True)) AND _language:(en))&fq=_indexname:(index_master_index)&facet=true&facet.field=computedarticlecategoryname_s&f.computedarticlecategoryname_s.facet.mincount=0&facet.field=computedresidencetypename_s&f.computedresidencetypename_s.facet.mincount=0&wt=xml

As you guys can see the stopped keyword is still there.

And since the indexing is doing its work properly, I don't have any for term indexed. The result is that the above query is always returning 0 results.

What am I doing wrong guys? Thanks in advance.

Answer 1

Query analysis (i.E. filtering stopwords) is done by Solr after a query has been sent. Sitecore's search.log log only logs the queries as they are sent to Solr. It is therefore correct to see your stop words in search.log.

To check how Solr actually analyzes a term, use the very helpful Analysis tool on your 'text_general' field:

https://mysolr/solr/#/mycore/analysis

The issue likely lies in the way you are combining the keyword predicates. Try running the logged query directly on the Solr UI and remove parts step by step to narrow down the issue.

Additionally, check the following:

• does your dynamicField *_t map to type="text_general" in managed-schema?
• is "for" acutally contained in your stopwords.txt?

Answer 2

Here is a more comprehensive Sitecore answer:

1. Find your dynamic field that you wish to query against in the Schema browser

Note the Dynamic Field and Type

2. Double confirm by viewing the dynamicField definition in managed-schema:

3. In my case, everything points to the type "text_en". Find it in managed-schema:

4. Locate the stopwords definition file

5. Pull down the configset, modify the stopwords file, reupload and then reload the collections

Also, in order to validate the changes:

1. Use the Analysis tool on your collection and be sure to select the TYPE, in my case the Type "text_en":

I can see that the ® is removed as I have added it as a stop word.