Working with chinese language with the SXA Search Component

Question

We have an issue, where we have content in multiple languages, and we are doing a basic keyword site search using the SXA Search Component. As far as I can tell (from the search.log), the SXA Search component looks at at a computed field called sxacontent:

<field type="Sitecore.XA.Foundation.Search.ComputedFields.AggregatedContent, Sitecore.XA.Foundation.Search" fieldName="sxacontent" returnType="textCollection">
   <mediaIndexing ref="contentSearch/indexConfigurations/defaultSolrIndexConfiguration/mediaIndexing"/>
</field>

This field is defined as a textcollection type (not text) - I'm not sure what is the difference. Regardless, textcollection's suffix is defined as {0}_txm:

<typeMatch type="System.Collections.Generic.List`1[System.String]" typeName="textCollection" fieldNameFormat="{0}_txm" multiValued="true" settingType="Sitecore.ContentSearch.SolrProvider.SolrSearchFieldConfiguration, Sitecore.ContentSearch.SolrProvider"/>

However, there is no definition in the dynamic field section of the solr managed-schema.xml file to define _txm fields for multiple languages. There is only the one entry:

<dynamicField name="*_txm" type="text_general" multiValued="true" indexed="true" stored="true"/>

I was expecting to see entries for *_txm_de, *_txm_zh, etc., like it exists for text fields, like *_t_de and other languages.

Given this, when we search for chinese language nothing shows up - we get no results.

We played with it a little bit and realized that the tokenizer for _txm is text_general, which is using the standard fieldtype, which doesn't work for chinese:

  <fieldType name="text_general" class="solr.TextField" positionIncrementGap="100" multiValued="false">
    <analyzer type="index">
      <tokenizer class="solr.StandardTokenizerFactory"/>
      <filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="true"/>
      <filter class="solr.LowerCaseFilterFactory"/>
    </analyzer>
    <analyzer type="query">
      <tokenizer class="solr.StandardTokenizerFactory"/>
      <filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="true"/>
      <filter class="solr.SynonymFilterFactory" expand="true" ignoreCase="true" synonyms="synonyms.txt"/>
      <filter class="solr.LowerCaseFilterFactory"/>
    </analyzer>
  </fieldType>

So we added a different fieldtype:

  <fieldType name="text_general_zh" class="solr.TextField" positionIncrementGap="100" multiValued="false">
    <analyzer type="index">
      <tokenizer class="org.apache.lucene.analysis.cn.smart.HMMChineseTokenizerFactory"/>
      <filter class="solr.StopFilterFactory" words="lang/stopwords_zh.txt" ignoreCase="true"/>
      <filter class="solr.PorterStemFilterFactory"/>
    </analyzer>
    <analyzer type="query">
        <tokenizer class="org.apache.lucene.analysis.cn.smart.HMMChineseTokenizerFactory" />
        <filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_zh.txt" />
        <filter class="solr.PorterStemFilterFactory"/>
    </analyzer>
  </fieldType>

...and added an entry for *_txm_zh to use that tokenizer:

<dynamicField name="*_txm_zh" type="text_general_zh" multiValued="true" indexed="true" stored="true"/>

But it doesn't work :(.

So, instead then we set the default *_txm to use the text_general_zh fieldtype

<dynamicField name="*_txm" type="text_general_zh" multiValued="true" indexed="true" stored="true"/>

Which made it work, but now all the languages are being processed by the Chinese tokenizer (HMMChineseTokenizerFactory) and that can't return the correct results in languages other than chinese.

We are trying to understand the best way to get the sxacontent field to be able to search in multiple languages, specifically chinese. What is the way to make this happen?

Answer 1

Out computes field sxacontent is defined as a textCollection because, in fact, it is an array of texts. This field is gathering content from the whole item and its fields. It is not only e.g.: content of a Title field but you will find much more texts from your item.

If you want to change something, for example:

• implement your own tokenizer in Solr and then use for the field
• do not tokenize certain field at all
• simply change the field type of existing fields

you need to do all of those by your own. We can't cover all of the customer cases in the default implementation.

You can also try with something like this: Detecting Languages During Indexing.

Answer 2

We had experienced the same issue with our last SXA project in Chinese. What we took as a workaroud is slightly modifing component-search-box.js to detect and split search keywords in Chinese with a space. When user search for "你好", the actual solr query will be like:

sxacontent_tm : (*你*) AND sxacontent_tm : (*好*) AND _language : (zh-CN).

So the search term will no longer be text array and be able to handle by default tokenizer. It's definitely not the best approach as search result like "你不好" will pop out, which is not likely the result that the user is expecting for, but we decided to stick to it, as there's just a couple of pages in Chinese for search on that site.

Hope it helps!