How to add charFilter to Solr Managed Schema from Sitecore 9?

Question

I need to add a charFilter to the Solr managed schema. I'm using Solr Cloud 6.6 and Sitecore 9.0.1. Sitecore uses a SchemaPopulateHelper class that generates xml updates to the managed schema. It essentially performs updates according to the following description, but does it using xml format instead of json:

https://lucene.apache.org/solr/guide/6_6/schema-api.html#SchemaAPI-MultipleCommandsinaSinglePOST

So a Sitecore managed field type update POST to the Solr API (reflected from Sitecore.ContentSearch.SolrProvider.Pipelines.PopulateSolrSchema.SchemaPopulateHelper) looks like this:

<replace-field-type>
  <name>text_general</name>
  <class>solr.TextField</class>
  <positionIncrementGap>100</positionIncrementGap>
  <multiValued>false</multiValued>
  <indexAnalyzer>
    <tokenizer>
      <class>solr.StandardTokenizerFactory</class>
    </tokenizer>
    <filters>
      <class>solr.WordDelimiterGraphFilterFactory</class>
      <catenateNumbers>1</catenateNumbers>
      <generateNumberParts>1</generateNumberParts>
      <splitOnCaseChange>1</splitOnCaseChange>
      <generateWordParts>1</generateWordParts>
      <catenateAll>0</catenateAll>
      <catenateWords>1</catenateWords>
      <splitOnNumerics>1</splitOnNumerics>
    </filters>
    <filters>
      <class>solr.LowerCaseFilterFactory</class>
    </filters>
  </indexAnalyzer>
  <queryAnalyzer>
    <tokenizer>
      <class>solr.StandardTokenizerFactory</class>
    </tokenizer>
    <filters>
      <class>solr.WordDelimiterGraphFilterFactory</class>
      <catenateNumbers>0</catenateNumbers>
      <generateNumberParts>1</generateNumberParts>
      <splitOnCaseChange>1</splitOnCaseChange>
      <generateWordParts>1</generateWordParts>
      <catenateAll>0</catenateAll>
      <catenateWords>0</catenateWords>
      <splitOnNumerics>1</splitOnNumerics>
    </filters>
    <filters>
      <class>solr.LowerCaseFilterFactory</class>
    </filters>
  </queryAnalyzer>
</replace-field-type>

So I essentially tried adding charFilters, like this:

...
<indexAnalyzer>
    <charFilters>
      <class>solr.PatternReplaceCharFilterFactory</class>
      <pattern>a</pattern>
      <replacement>b</replacement>
    </charFilters>
    <tokenizer>
...

This doesn't work, and I've tried a few variants of this. The tricky thing is that Solr accepts the update request, without any errors in the Sitecore log. There are no errors in the Solr logs either. The only way I can see that this fails is to make other changes to the schema at the same time and see that those are not stored either. If I remove the charFilter and make other changes to the field type, I can see that Solr picks this up and stores them into the managed schema.

Any ideas on how this update should be written or how I could debug this?

Answer 1

It turned out this is caused by a design flaw in Sitecore...

The XML that Sitecore creates internally is never posted to Solr. Instead, Sitecore converts the internal XML representation to JSON before posting it to Solr, using the SchemaCommand class where it serializes the XElement list with JsonConvert.

The Solr schema REST api requires charFilters to be an array and when having only one charFilter in the XML representation, there is no way that the json conversion would know that this should be represented as an array.

So essentially, if I add two charFilters, like the filter I want, and a second dummy filter that replaces for example "a" with "a", the schema update works as expected.

This design flaw also means that it isn't possible to create a field type with only one Filter, though that scenario is probably quite rare.