Solr tokenization to split on semicolon character

Question

How do I go about configuring fields within a Solr index to be tokenized based on a semicolon ";" delimiter?

Initially, I had fields configured as such:

<field
   fieldName="example"
   storageType="YES"
   indexType="UN_TOKENIZED"
   vectorType="NO"
   boost="1f"
   returnType="text"
   settingType="Sitecore.ContentSearch.SolrProvider.SolrSearchFieldConfiguration, Sitecore.ContentSearch.SolrProvider"/>

which causes the field to be tokenised on individual words.

Then I changed

returnType="string"

which causes the whole field to be treated as a single value.

I want it to tokenize using a semicolon delimiter. How do I configure the field to do this?

Answer 1

You can use solr.PatternTokenizerFactory tokenizer to achieve this

Define a new fieldtype in solr schema

<fieldType name="semicolonDelimited" class="solr.TextField">
  <analyzer>      
    <tokenizer class="solr.PatternTokenizerFactory" pattern=";" />
  </analyzer>
</fieldType> 

and then define the field which uses semicolonDelimited fieldtype

<field name="content_custom" type="semicolonDelimited"/>

Test Scenerio

1. Create a sample solr core
2. Define FieldType and Field as shown below

<fieldType name="semicolonDelimited" class="solr.TextField">
  <analyzer>      
    <tokenizer class="solr.PatternTokenizerFactory" pattern=";" />
    <filter class="solr.LowerCaseFilterFactory"/>
  </analyzer>
</fieldType>  
<field name="content_custom" type="semicolonDelimited"/>

3. Restart Solr Service
4. Import test data to sample core

Sample JSON

{
  "id": 1,
  "content_custom": "Ten;10"
},
{
  "id": 2,
  "content_custom": "Eleven;11"
},
{
  "id": 3,
  "content_custom": "Twelve;12"
},
{
  "id": 4,
  "content_custom": "Thirteen;13"
}

Copy JSON and upload it to Solr

5. Now Search for content_custom:t*n and then it will show two results

Answer 2

If you were to do it with a ComputedIndex field. it would be like this. But maybe you can do it with just solr configs.

using System.Collections.Generic;

namespace Sitecore.Foundation.Indexing.Infrastructure.Fields
{
    using Sitecore.ContentSearch;
    using Sitecore.ContentSearch.ComputedFields;
    using Sitecore.Foundation.SitecoreExtensions.Extensions;

    public class SemicolonContentComputedField : IComputedIndexField
    {
        public string FieldName { get; set; }
        public string ReturnType { get; set; }

        public object ComputeFieldValue(IIndexable indexable)
        {
            if (!(indexable is SitecoreIndexableItem indexItem))
            {
                return null;
            }

            var item = indexItem.Item;
            if (item == null) return null;

            if (item.IsDerived(Constants.Templates.PhpContent))
            {
                var results = new List<string>();
                var splits = item[Templates.PhpContent.Fields.SemiColonField].Split(';');
                foreach (var split in splits)
                {
                    results.Add(split);
                }

                return results;
            }

            return null;
        }
    }
}

Then the config would be stringcollection

<?xml version="1.0" encoding="utf-8"?>
<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/" xmlns:set="http://www.sitecore.net/xmlconfig/set/">
    <sitecore>
        <contentSearch>
            <indexConfigurations>
                <defaultSolrIndexConfiguration type="Sitecore.ContentSearch.SolrProvider.SolrIndexConfiguration, Sitecore.ContentSearch.SolrProvider">
                    <fieldMap type="Sitecore.ContentSearch.SolrProvider.SolrFieldMap, Sitecore.ContentSearch.SolrProvider">
                        <fieldNames hint="raw:AddFieldByFieldName">
                            <field fieldName="semicolon" returnType="stringCollection" />
                        </fieldNames>
                    </fieldMap>
                    <documentOptions type="Sitecore.ContentSearch.SolrProvider.SolrDocumentBuilderOptions, Sitecore.ContentSearch.SolrProvider">
                        <fields hint="raw:AddComputedIndexField">
                            <field fieldName="semicolon" storageType="no" indexType="untokenized">Sitecore.Foundation.Indexing.Infrastructure.Fields.SemicolonContentComputedField, Sitecore.Foundation.Indexing</field>
                        </fields>
                    </documentOptions>
                </defaultSolrIndexConfiguration>
            </indexConfigurations>
        </contentSearch>
    </sitecore>
</configuration>