Solr Search Issue for Chinese Language in Sitecore 9.0.1

Question

I am using Sitecore 9.0.1 with SXA 1.7 and Solr version 6.6.2. we have a requirement to search the chineses content so i have created some chinese content and in the page i have added the components

1. Added a SXA Search box component and Search Result component on the page.

2.Specified the Scope query to search on to.

But when i search for chinese word it is not giving the results for it but i add space between the letters it is giving the results for it.

Without Space result:

With Space between the letters:

In Solr schema i have added the below set of codes:

1.

<dynamicField name="*_txm_zh" type="text_general_zh" multiValued="true" indexed="true" stored="true"/>

In Solr config i have added the below line of code:

<lib dir="${solr.install.dir:../../../..}/contrib/analysis-extras/lucene-libs" regex=".*\.jar" />

Indexed data in solr :

Can anyone please suggest what i have to do in order to search without space and get the results.

Thanks

Answer 1

The problem indicates that you'r Solr is indexing letters in this case differently. You need to setup new custom field on Solr schema and define what tokenizer to use. Tokenizer logic is responsible how to index words. For example in Finnish language "e-lasku" is one word but it was indexed as "e" and "lasku". I had to create custom field with different tokenizer. If you look at the schema.xml file in Solr Core you can find "text_general" field is defined as like this:

<fieldType name="text_general" class="solr.TextField" positionIncrementGap="100" multiValued="true">
    <analyzer type="index">
      <tokenizer class="solr.StandardTokenizerFactory" />
      <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" />
      <!-- in this example, we will only use synonyms at query time
        <filter class="solr.SynonymFilterFactory" synonyms="index_synonyms.txt" ignoreCase="true" expand="false"/>
        -->
      <filter class="solr.LowerCaseFilterFactory" />
    </analyzer>
    <analyzer type="query">
      <tokenizer class="solr.StandardTokenizerFactory" />
      <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" />
      <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true" />
      <filter class="solr.LowerCaseFilterFactory" />
    </analyzer>
  </fieldType>

On the third row is tokenizer defined as StandardTokenizerFactory. That split words when "-" is on the word. This is most likely the reason also why Chinese letters are split. You need to create your own Solr Core field and use different tokenizer. Or you can remove this tokenizer on "text_general_zh" and replace it with this:

<tokenizer class="solr.WhitespaceTokenizerFactory" />

If you want to define new field you can do it like this. Add new fieldType:

<fieldType name="text_general_zh_custom" class="solr.TextField" positionIncrementGap="100" multiValued="false">
     <analyzer type="index">
        <tokenizer class="solr.WhitespaceTokenizerFactory" />
      <filter class="solr.LowerCaseFilterFactory" />
      </analyzer>
      <analyzer type="query">
        <tokenizer class="solr.WhitespaceTokenizerFactory" />
        <filter class="solr.LowerCaseFilterFactory" />
      </analyzer>
   </fieldType>   

And also add this line in the same file where rest of these are:

<field name="content_custom" type="text_general_custom" indexed="true" stored="true"/>

Lastly add a custom solr index on Sitecore and make code use that field. Hopefully this resolves your problem.