2

I'm having some fun with Solr and here is my current issue.

I've applied some stop words using the stopwords.txt file and rebuilt my indexes. Everything looks fine here, I can see that those stop words are no longer indexed:

enter image description here

All that being said Query Time is my issue.

Firstly, here is my text_general field type config on managed-schema:

<fieldType name="text_general" class="solr.TextField" positionIncrementGap="100" multiValued="false">
    <analyzer type="index">
      <tokenizer class="solr.StandardTokenizerFactory"/>
      <filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="true"/>
      <filter class="solr.LowerCaseFilterFactory"/>
    </analyzer>
    <analyzer type="query">
      <tokenizer class="solr.StandardTokenizerFactory"/>
      <filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="true"/>
      <filter class="solr.SynonymFilterFactory" expand="true" ignoreCase="true" synonyms="synonyms.txt"/>
      <filter class="solr.LowerCaseFilterFactory"/>
    </analyzer>
</fieldType>

Since the StopFilterFactory is there on both Index and Query time I'm wondering why queries with stop words are keeping those words during execution.

Let's say I'm quering Designed for you.

Since for is a stopword my query should really be Designed you.

Here is the code handling the query:

var keywordsFilter = PredicateBuilder.True<UnifiedSearchResultItemWithLocalDatasource>();
            foreach (var keyword in keywords)
            {
                var keywordFilter = PredicateBuilder.False<UnifiedSearchResultItemWithLocalDatasource>();
                keywordFilter = keywordFilter.Or(item => item.UnifiedTitle.Like(keyword, 0.8f).Boost(8f))
                    .Or(item => item.UnifiedTitle.StartsWith(keyword).Boost(8f))
                    .Or(item => item.UnifiedSiteSection.Like(keyword, 0.8f).Boost(4f))
                    .Or(item => item.UnifiedSiteSection.StartsWith(keyword).Boost(4f))
                    .Or(item => item.UnifiedDescription.Like(keyword, 0.8f).Boost(2f))
                    .Or(item => item.UnifiedDescription.StartsWith(keyword).Boost(2f));

                keywordsFilter = keywordsFilter.And(keywordFilter);
            }

And here is the Search log with the generated Solr query:

16776 17:32:29 INFO Solr Query - ?q=((-unifiedtitle_t:("") unifiedtitle_t:[* TO ]) AND ((((unifiedtitle_t:(designed~0.8))^8 OR (unifiedtitle_t:(designed))^8 OR (unifiedsitesection_t:(designed~0.8))^4 OR (unifiedsitesection_t:(designed*))^4 OR (unifieddescription_t:(designed~0.8))^2 OR (unifieddescription_t:(designed*))^2) AND ((unifiedtitle_t:(for~0.8))^8 OR (unifiedtitle_t:(for*))^8 OR (unifiedsitesection_t:(for~0.8))^4 OR (unifiedsitesection_t:(for*))^4 OR (unifieddescription_t:(for~0.8))^2 OR (unifieddescription_t:(for*))^2)) AND ((unifiedtitle_t:(you~0.8))^8 OR (unifiedtitle_t:(you*))^8 OR (unifiedsitesection_t:(you~0.8))^4 OR (unifiedsitesection_t:(you*))^4 OR (unifieddescription_t:(you~0.8))^2 OR (unifieddescription_t:(you*))^2)))&start=0&rows=10&fl=*,score&fq=((_latestversion:(True) AND has_presentation_b:(True)) AND _language:(en))&fq=_indexname:(index_master_index)&facet=true&facet.field=computedarticlecategoryname_s&f.computedarticlecategoryname_s.facet.mincount=0&facet.field=computedresidencetypename_s&f.computedresidencetypename_s.facet.mincount=0&wt=xml

As you guys can see the stopped keyword is still there.

And since the indexing is doing its work properly, I don't have any for term indexed. The result is that the above query is always returning 0 results.

What am I doing wrong guys? Thanks in advance.

  • 1
    I know it's a pretty lame suggestion but can you try Reloading the index and try. – Sumit Upadhyay Oct 29 at 13:23
  • You mean restart the Solr service and/or rebuilding the index? Already did and still the same, thanks anyway. – Hugo Santos Oct 29 at 13:56
  • No, in the Core Admin there is an option called "Reload". Try that – Sumit Upadhyay Oct 29 at 14:01
  • Tried that, same results. Thanks – Hugo Santos Oct 29 at 14:07
2

Query analysis (i.E. filtering stopwords) is done by Solr after a query has been sent. Sitecore's search.log log only logs the queries as they are sent to Solr. It is therefore correct to see your stop words in search.log.

To check how Solr actually analyzes a term, use the very helpful Analysis tool on your 'text_general' field:

https://mysolr/solr/#/mycore/analysis

The issue likely lies in the way you are combining the keyword predicates. Try running the logged query directly on the Solr UI and remove parts step by step to narrow down the issue.

Additionally, check the following:

  • does your dynamicField *_t map to type="text_general" in managed-schema?
  • is "for" acutally contained in your stopwords.txt?
  • You are totally right Mark, thanks a lot. However, I'm not sure if the ANDs are the issue here. I edited my original query down to ((unifieddescription_t:(for~0.8))) and tried to run it agains Solr directly. Still, no results. – Hugo Santos Oct 29 at 14:57
  • Btw here is the parsedString when running in debug mode: "parsedquery":"unifieddescription_t:for~0", – Hugo Santos Oct 29 at 14:58
  • Should the above query behave just like this one (if for is a stop word)? unifieddescription_t:(*) – Hugo Santos Oct 29 at 15:04
  • 1
    Querying for a stop word returns 0 results. I just tested this. – Mark Lowe Oct 29 at 15:07
  • Alright, so maybe I had wrong expectations related to this subject. So basically if I still want to use stop words I'll have to change my keyword AND keyword AND keyword to keyword OR keyword OR keyword, otherwise a query with stop words will always have 0 results. Is this assumption right? – Hugo Santos Oct 29 at 15:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.