Using alternate ContentSearch analyzers

Question

I am looking into using some of the alternate analyzers provided in the Sitecore.ContentSearch.LuceneProvider.Analyzers namespace.

Specifically what I would like to achieve is to:

• Create copies of the _content field and apply different analyzer to them (one stemmed and another ngrammed).
• Be able to control the analyzer that is used at query time so that I can search with either a stemmer or an ngram analyzer.

So, using Sitecore.ContentSearch.LuceneProvider.Analyzers.StandardAnalyzerWithStemming and Sitecore.ContentSearch.LuceneProvider.Analyzers.NGramAnalyzer are of interest.

However, trying to get them implemented in the config files is another thing. I am experiencing crashes with no log/stack trace any time of indexing and can't seem to be able to get specific fields included in the index but only analyzed by the LowerCaseKeywordAnalyzer (ex. things like GUIDs which a stemmer wreaks havoc on).

I have (I think) scoured the web and have a few bits and pieces of information but nothing concrete for these analyzers. Aside from personal experience, if anyone knows of some good Sitecore documentation on the analyzers or on index configuration files, anything would be helpful. Thanks.

Answer 1

It should be possible, but probably not that simple - at least not if you want to use a different analyzer at search time.

Please note that I haven't tested any of the stuff below.

Changing the analyzer (both indexin and searching)

You should be able to change the analyzer of the _content field like this:

<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
  <sitecore>
    <contentSearch>
      <indexConfigurations>
        <defaultLuceneIndexConfiguration>
          <fieldMap type="Sitecore.ContentSearch.FieldMap, Sitecore.ContentSearch">
            <fieldNames hint="raw:AddFieldByFieldName">
              <field fieldName="_content" storageType="NO" indexType="TOKENIZED" vectorType="NO" boost="1f" type="System.String"  settingType="Sitecore.ContentSearch.LuceneProvider.LuceneSearchFieldConfiguration, Sitecore.ContentSearch.LuceneProvider">
                <analyzer type="Sitecore.ContentSearch.LuceneProvider.Analyzers.StandardAnalyzerWithStemming, Sitecore.ContentSearch.LuceneProvider">
                  <param hint="version">Lucene_30</param>
                </analyzer>
              </field>
            </fieldNames>
          </fieldMap>
        </defaultLuceneIndexConfiguration>
      </indexConfigurations>
    </contentSearch>
  </sitecore>
</configuration>

This should then use the specified analyzer for both indexing and searching this field.

Copying the _content field with different analyzer

The _content field is a special field added by Sitecore at index time.

If you want to index _content more than once, using different analyzers, I think you would need to extend Sitecore.ContentSearch.LuceneProvider.LuceneDocumentBuilder and override the method void AddField(IIndexableDataField field).

object obj = fieldConfiguration1.FormatForWriting(fieldValue);
float boost = BoostingManager.ResolveFieldBoosting(field);
if (IndexOperationsHelper.IsTextField(field))
{
    LuceneSearchFieldConfiguration fieldConfiguration2 = this.Index.Configuration.FieldMap.GetFieldConfiguration("_content") as LuceneSearchFieldConfiguration;
    this.AddField("_content", obj, fieldConfiguration2 ?? this.defaultTextField, 0.0f);
}
this.AddField(name, obj, fieldSettings, boost);

In that method you could then add your own _content_ngram or something like that and control the used analyzer in the field configuration as shown above.

Using another analyzer at search time

If you want to use another analyzer at search time than the one specified on the field configuration, then you could probably utilize execution contexts by implementing Sitecore.ContentSearch.Linq.Common.IExecutionContext. The implementations themselves do not need any logic.

public class NGramExecutionContext : IExecutionContext 
{
}

You would then need to create a custom analyzer which uses a specific analyzer depending on the field name.

<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
  <sitecore>
    <contentSearch>
      <indexConfigurations>
        <defaultLuceneIndexConfiguration>
          <analyzer type="Sitecore.ContentSearch.LuceneProvider.Analyzers.PerExecutionContextAnalyzer, Sitecore.ContentSearch.LuceneProvider">
            <param desc="map" type="System.Collections.Generic.List`1[[Sitecore.ContentSearch.LuceneProvider.Analyzers.PerExecutionContextAnalyzerMapEntry, Sitecore.ContentSearch.LuceneProvider]]">
              <map hint="list:Add">
                <mapEntry type="Sitecore.ContentSearch.LuceneProvider.Analyzers.PerExecutionContextAnalyzerMapEntry, Sitecore.ContentSearch.LuceneProvider">
                  <!-- Your custom execution context -->
                  <param hint="executionContext" type="YourNamespace.NGramExecutionContext, YourAssembly">
                    <!-- Any parameters for your execution context constructor -->
                  </param>
                  <!-- Your custom analyzer -->
                  <param desc="analyzer" type="YourNamespace.CustomAnalyzer, YourAssembly">
                    <!-- Any paramters for your custom analyzer constructor -->
                  </param>
                </mapEntry>
              </map>
            </param>
          </analyzer>
        </defaultLuceneIndexConfiguration>
      </indexConfigurations>
    </contentSearch>
  </sitecore>
</configuration>

How to implement the custom analyzer I'll leave up to you. You could take a look at how Sitecore.ContentSearch.LuceneProvider.Analyzers.DefaultPerFieldAnalyzer works to get some inspiration.