Handling special characters in SOLR search for sitecore 9.1

Question

For a Dutch company, we implemented Sitecore SOLR search (Sitecore 9.1 and SOLR 7.2.1 ) with a requirement for the user to be able to search words with special Dutch characters for example if the user searches for financiel he should be able to find => financiël. I added the following to the managed schema of the SOLR.

<fieldType name="text_general" class="solr.TextField" positionIncrementGap="100" multiValued="false">
<analyzer type="index">
    <filter class="**solr.ASCIIFoldingFilterFactory**"/>
 <charFilter class="**solr.MappingCharFilterFactory**" mapping="**mapping-FoldToASCII.txt**"/>
  <tokenizer class="solr.StandardTokenizerFactory"/>
  <filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="true"/>
  <filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
    <filter class="solr.ASCIIFoldingFilterFactory"/>

This is my custom index managed schema

<?xml version="1.0" encoding="utf-8" ?>
<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
 <sitecore>
  <contentSearch>
    <customSolrManagedSchema>
     <commands applyToIndex="sitecore_content_index">
      <add-or-replace-field-type>
        <name>ngram</name>
        <class>solr.TextField</class>
        <positionIncrementGap>100</positionIncrementGap>
        <indexAnalyzer>
          <tokenizer>
            <class>solr.StandardTokenizerFactory</class>
          </tokenizer>
          <filters>
            <class>solr.LowerCaseFilterFactory</class>
          </filters>
          <filters>
            <class>solr.ASCIIFoldingFilterFactory</class>
            <preserveOriginal>false</preserveOriginal>
          </filters>
              <charFilter class="solr.MappingCharFilterFactory" mapping="mapping-FoldToASCII.txt"/>
          <filters>
            <class>solr.NGramFilterFactory</class>
            <minGramSize>2</minGramSize>
            <minGramSize>35</minGramSize>
          </filters>
       
        </indexAnalyzer>
        <queryAnalyzer>
          <type>query</type>
          <tokenizer>
            <class>solr.StandardTokenizerFactory</class>
          </tokenizer>
          <filters>
            <class>solr.LowerCaseFilterFactory</class>
          </filters>
          <filters>
            <class>solr.ASCIIFoldingFilterFactory</class>
            <preserveOriginal>false</preserveOriginal>
          </filters>
            <charFilter class="solr.MappingCharFilterFactory" mapping="mapping-FoldToASCII.txt"/>
        </queryAnalyzer>
      </add-or-replace-field-type>

      <add-dynamic-field>
        <name>*_ngram</name>
        <type>ngram</type>
        <indexed>true</indexed>
        <stored>true</stored>
        <termVectors>true</termVectors>
        <termPositions>true</termPositions>
        <termOffsets>true</termOffsets>
      </add-dynamic-field>
      <add-dynamic-field>
        <name>*_ngramm</name>
        <type>ngram</type>
        <indexed>true</indexed>
        <stored>true</stored>
        <termVectors>true</termVectors>
        <termPositions>true</termPositions>
        <termOffsets>true</termOffsets>
        <multiValued>true</multiValued>
      </add-dynamic-field>
    </commands>
  </customSolrManagedSchema>
</contentSearch>
<pipelines>
  <contentSearch.PopulateSolrSchema>
    <!-- Processor for additional SOLR schema -->
    <processor type="SAA.Feature.Search.Fields.CustomSchemaPopulator, SAA.Feature.Search"               patch:after="processor[@type='Sitecore.ContentSearch.SolrProvider.Pipelines.PopulateSolrSchema.PopulateFields, Sitecore.ContentSearch.SolrProvider']"/>
  </contentSearch.PopulateSolrSchema>
</pipelines>

For some reason after adding the following line <charFilter class="solr.MappingCharFilterFactory" mapping="mapping-FoldToASCII.txt"/> to the managed schema as described in some of the tutorials , the index was no longer visible in the Sitecore Index manager. If the line is removed the index is visible and some of the fields are indexed, correctly, but some of them are not especially Richtext fields. I am also not sure where is the location of the file described <charFilter class="solr.MappingCharFilterFactory" mapping="mapping-FoldToASCII.txt"/> in that line. I also tried reorganizing the filters in the schema without any success. Is there any working solution with the following versions of Sitecore 9.1 and SOLR 7.2.1? Thanks in advance!

Answer 1

1.Make sure that you are using preserveOriginal="true" for solr.ASCIIFoldingFilterFactory filter

<fieldType name="text_general" class="solr.TextField" positionIncrementGap="100">
  <analyzer type="index">
    <tokenizer  class="solr.StandardTokenizerFactory"/>
    <filter     class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt"/>
    <filter     class="solr.LowerCaseFilterFactory"/>        
    <filter     class="solr.ASCIIFoldingFilterFactory" preserveOriginal="true"/>
    <charFilter class="solr.MappingCharFilterFactory" mapping="mapping-FoldToASCII.txt"/>        
  </analyzer>
  <analyzer type="query">
    <tokenizer class="solr.StandardTokenizerFactory"/>
    <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" />
    <filter class="solr.SynonymGraphFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
    <filter class="solr.LowerCaseFilterFactory"/>
    <filter class="solr.ASCIIFoldingFilterFactory"/>
  </analyzer>
</fieldType>

2. Make sure mapping-FoldToASCII.txt is there in the conf folder

3. Before proceeding to testing part, make sure you re-index sample item

4. Then test changes from Solr admin UI

5. Then test it from actual search page

Here is the sample test result -