3

For a Dutch company, we implemented Sitecore SOLR search (Sitecore 9.1 and SOLR 7.2.1 ) with a requirement for the user to be able to search words with special Dutch characters for example if the user searches for financiel he should be able to find => financiël. I added the following to the managed schema of the SOLR.

<fieldType name="text_general" class="solr.TextField" positionIncrementGap="100" multiValued="false">
<analyzer type="index">
    <filter class="**solr.ASCIIFoldingFilterFactory**"/>
 <charFilter class="**solr.MappingCharFilterFactory**" mapping="**mapping-FoldToASCII.txt**"/>
  <tokenizer class="solr.StandardTokenizerFactory"/>
  <filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="true"/>
  <filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
    <filter class="solr.ASCIIFoldingFilterFactory"/>

This is my custom index managed schema

<?xml version="1.0" encoding="utf-8" ?>
<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
 <sitecore>
  <contentSearch>
    <customSolrManagedSchema>
     <commands applyToIndex="sitecore_content_index">
      <add-or-replace-field-type>
        <name>ngram</name>
        <class>solr.TextField</class>
        <positionIncrementGap>100</positionIncrementGap>
        <indexAnalyzer>
          <tokenizer>
            <class>solr.StandardTokenizerFactory</class>
          </tokenizer>
          <filters>
            <class>solr.LowerCaseFilterFactory</class>
          </filters>
          <filters>
            <class>solr.ASCIIFoldingFilterFactory</class>
            <preserveOriginal>false</preserveOriginal>
          </filters>
              <charFilter class="solr.MappingCharFilterFactory" mapping="mapping-FoldToASCII.txt"/>
          <filters>
            <class>solr.NGramFilterFactory</class>
            <minGramSize>2</minGramSize>
            <minGramSize>35</minGramSize>
          </filters>
       
        </indexAnalyzer>
        <queryAnalyzer>
          <type>query</type>
          <tokenizer>
            <class>solr.StandardTokenizerFactory</class>
          </tokenizer>
          <filters>
            <class>solr.LowerCaseFilterFactory</class>
          </filters>
          <filters>
            <class>solr.ASCIIFoldingFilterFactory</class>
            <preserveOriginal>false</preserveOriginal>
          </filters>
            <charFilter class="solr.MappingCharFilterFactory" mapping="mapping-FoldToASCII.txt"/>
        </queryAnalyzer>
      </add-or-replace-field-type>

      <add-dynamic-field>
        <name>*_ngram</name>
        <type>ngram</type>
        <indexed>true</indexed>
        <stored>true</stored>
        <termVectors>true</termVectors>
        <termPositions>true</termPositions>
        <termOffsets>true</termOffsets>
      </add-dynamic-field>
      <add-dynamic-field>
        <name>*_ngramm</name>
        <type>ngram</type>
        <indexed>true</indexed>
        <stored>true</stored>
        <termVectors>true</termVectors>
        <termPositions>true</termPositions>
        <termOffsets>true</termOffsets>
        <multiValued>true</multiValued>
      </add-dynamic-field>
    </commands>
  </customSolrManagedSchema>
</contentSearch>
<pipelines>
  <contentSearch.PopulateSolrSchema>
    <!-- Processor for additional SOLR schema -->
    <processor type="SAA.Feature.Search.Fields.CustomSchemaPopulator, SAA.Feature.Search"               patch:after="processor[@type='Sitecore.ContentSearch.SolrProvider.Pipelines.PopulateSolrSchema.PopulateFields, Sitecore.ContentSearch.SolrProvider']"/>
  </contentSearch.PopulateSolrSchema>
</pipelines>

For some reason after adding the following line <charFilter class="solr.MappingCharFilterFactory" mapping="mapping-FoldToASCII.txt"/> to the managed schema as described in some of the tutorials , the index was no longer visible in the Sitecore Index manager. If the line is removed the index is visible and some of the fields are indexed, correctly, but some of them are not especially Richtext fields. I am also not sure where is the location of the file described <charFilter class="solr.MappingCharFilterFactory" mapping="mapping-FoldToASCII.txt"/> in that line. I also tried reorganizing the filters in the schema without any success. Is there any working solution with the following versions of Sitecore 9.1 and SOLR 7.2.1? Thanks in advance!

  • After adding the fields in xml , please restart SOLR service and check the SOLR dashboard -> logging for any errors – Abhay Dhar Aug 25 at 12:00
  • Hi Abhay, I tried and I found that there was an error about => could not find the mapping-FoldToASCII.txt file, which I added in the index directory, so the error was no longer there, but the problem with special characters is still there. Thanks for the quick reply ! – Ange Chakurov Aug 25 at 14:33
  • Pl add more details to the question , Does sitecore index manager pick up the new Indexes ? – Abhay Dhar Aug 25 at 17:53
  • Yes, it does. The index and the search are working except the part with the special chars. – Ange Chakurov Aug 26 at 6:34
  • I can see this file 'mapping-FoldToASCII.txt' in \server\solr\configsets\sample_techproducts_configs\conf – Paulson Maclean Aug 30 at 14:52
2

1.Make sure that you are using preserveOriginal="true" for solr.ASCIIFoldingFilterFactory filter

<fieldType name="text_general" class="solr.TextField" positionIncrementGap="100">
  <analyzer type="index">
    <tokenizer  class="solr.StandardTokenizerFactory"/>
    <filter     class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt"/>
    <filter     class="solr.LowerCaseFilterFactory"/>        
    <filter     class="solr.ASCIIFoldingFilterFactory" preserveOriginal="true"/>
    <charFilter class="solr.MappingCharFilterFactory" mapping="mapping-FoldToASCII.txt"/>        
  </analyzer>
  <analyzer type="query">
    <tokenizer class="solr.StandardTokenizerFactory"/>
    <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" />
    <filter class="solr.SynonymGraphFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
    <filter class="solr.LowerCaseFilterFactory"/>
    <filter class="solr.ASCIIFoldingFilterFactory"/>
  </analyzer>
</fieldType>
  1. Make sure mapping-FoldToASCII.txt is there in the conf folder

  2. Before proceeding to testing part, make sure you re-index sample item

  3. Then test changes from Solr admin UI

  4. Then test it from actual search page

Here is the sample test result -

enter image description here

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.