0

I am using Sitecore 9.0.1 with SXA 1.7 and Solr version 6.6.2. we have a requirement to search the chineses content so i have created some chinese content and in the page i have added the components

  1. Added a SXA Search box component and Search Result component on the page.

2.Specified the Scope query to search on to.

But when i search for chinese word it is not giving the results for it but i add space between the letters it is giving the results for it.

Without Space result: enter image description here

With Space between the letters:

enter image description here

In Solr schema i have added the below set of codes:

1.

<dynamicField name="*_txm_zh" type="text_general_zh" multiValued="true" indexed="true" stored="true"/>

In Solr config i have added the below line of code:

<lib dir="${solr.install.dir:../../../..}/contrib/analysis-extras/lucene-libs" regex=".*\.jar" />

Indexed data in solr :

enter image description here

Can anyone please suggest what i have to do in order to search without space and get the results.

Thanks

4
  • The problem might indicate that when Solr is indexing chinese words it is adding space between them. Check from Solr is it indexing like that. There is a tool that you can add words and it shows you how it will index it. Add chinese words and check if it adds spaces. Apr 8 at 19:21
  • @Jonkoivula no in solr there is no space between the words i have update the scrnshot in the question
    – Vinaya
    Apr 9 at 5:19
  • This isn't the correct place to check. This shows you the documents what are indexed, it doesnt actually show how it's indexed in Solr. Solr has different rules how to break words. Go to Solr in your browser, choose the correct Solr Core, then go to Analysis page. selected Analyse FieldName text_general_zh from dropdown list and add your chinese letters to Field Value textfield and click Analyse Values. Post a screenshot from that, it will show you if it breaks those two chinese letters. Apr 9 at 15:08
  • @Jon Koivula updated the screenshot of solr as you suggested..yes it is dividing the letters .Can you please suggest how can we resolve this issue
    – Vinaya
    Apr 12 at 8:50
0

The problem indicates that you'r Solr is indexing letters in this case differently. You need to setup new custom field on Solr schema and define what tokenizer to use. Tokenizer logic is responsible how to index words. For example in Finnish language "e-lasku" is one word but it was indexed as "e" and "lasku". I had to create custom field with different tokenizer. If you look at the schema.xml file in Solr Core you can find "text_general" field is defined as like this:

<fieldType name="text_general" class="solr.TextField" positionIncrementGap="100" multiValued="true">
    <analyzer type="index">
      <tokenizer class="solr.StandardTokenizerFactory" />
      <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" />
      <!-- in this example, we will only use synonyms at query time
        <filter class="solr.SynonymFilterFactory" synonyms="index_synonyms.txt" ignoreCase="true" expand="false"/>
        -->
      <filter class="solr.LowerCaseFilterFactory" />
    </analyzer>
    <analyzer type="query">
      <tokenizer class="solr.StandardTokenizerFactory" />
      <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" />
      <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true" />
      <filter class="solr.LowerCaseFilterFactory" />
    </analyzer>
  </fieldType>

On the third row is tokenizer defined as StandardTokenizerFactory. That split words when "-" is on the word. This is most likely the reason also why Chinese letters are split. You need to create your own Solr Core field and use different tokenizer. Or you can remove this tokenizer on "text_general_zh" and replace it with this:

<tokenizer class="solr.WhitespaceTokenizerFactory" />

If you want to define new field you can do it like this. Add new fieldType:

<fieldType name="text_general_zh_custom" class="solr.TextField" positionIncrementGap="100" multiValued="false">
     <analyzer type="index">
        <tokenizer class="solr.WhitespaceTokenizerFactory" />
      <filter class="solr.LowerCaseFilterFactory" />
      </analyzer>
      <analyzer type="query">
        <tokenizer class="solr.WhitespaceTokenizerFactory" />
        <filter class="solr.LowerCaseFilterFactory" />
      </analyzer>
   </fieldType>   

And also add this line in the same file where rest of these are:

<field name="content_custom" type="text_general_custom" indexed="true" stored="true"/>

Lastly add a custom solr index on Sitecore and make code use that field. Hopefully this resolves your problem.

2
  • is it necessary to create a solr core ?
    – Vinaya
    Apr 12 at 14:11
  • no its not, you can just change schema.xml file. Change tokenizer and re-index your solr core. Remember to restart Solr after changing schema. Be aware when you change StandardTokenizer it changes how it indexes words. You can read more from Solr documentation how it does it. Apr 12 at 15:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.