Solr and Tokenizers: how to tokenize specific indexes using Solr 4.10.4

Question

I'm on Solr 4.10.4, Sitecore 8.1 160519.

I finally managed to convert my Lucene custom index configuration to Solr, but some indexes are not tokenized. The custom index "parsed_answers" on Lucene is defined as follows:

<field fieldName="parsed_answers" storageType="yes" indexType="tokenized" />

while on Solr is defined like so:

<field fieldName="parsed_answers" returnType="stringCollection" storageType="yes" indexType="tokenized" multiValued="true">

My custom code returns a lengthy string. Lucene breaks it down into individual words and stores each word in the index individually. Solr simply takes the entire text blurb and indexes it. This is not desirable.

Question

I have read about tokenizers in Solr:

<fieldType name="text" class="solr.TextField">
  <analyzer>
    <tokenizer class="solr.StandardTokenizerFactory"/>
  </analyzer>
</fieldType>

I could not, however, find an example that I can use in my case yet - do I have to define a new typeMatch that is tokenized, or can I tokenize an existing one?

Any help is appreciated.

Answer 1

On https://lucene.apache.org/solr/guide/6_6/field-types-included-with-solr.html#field-types-included-with-solr you can find a description of the Solr field types. (or https://archive.apache.org/dist/lucene/solr/ref-guide/apache-solr-ref-guide-4.10.pdf for Solr 4)

For "string" they mention:

String (UTF-8 encoded string or Unicode). Strings are intended for small fields and are not tokenized or analyzed in any way. They have a hard limit of slightly less than 32K.

As your stringCollection is a collection of strings, they will not be tokenized. To get that functionality, you need to use text - or in your multiple value case a textCollection.

Must admit that I never used a textCollection myself, but based on the docs that should work.

Answer 2

As it turns out, the only way of achieving this with Solr 4.10.4 is by modifying the schema.xml file of the Solr core.

What I did was as follows:

1) In the Sitecore custom configuration I defined a < typeMatch > called "customStringCol":

<typeMatch typeName="customStringCol" type="System.Collections.Generic.List`1[System.String]" fieldNameFormat="{0}_csm" multiValued="true" indexed="true" settingType="Sitecore.ContentSearch.SolrProvider.SolrSearchFieldConfiguration, Sitecore.ContentSearch.SolrProvider"/>

2) I then created the same custom fieldType "customStringCol" in the schema.xml file and added a Tokenizer to it:

<fieldType name="customStringCol" class="solr.TextField" sortMissingLast="true">
  <analyzer type="index">
    <tokenizer class="solr.StandardTokenizerFactory" />
    <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" />
    <filter class="solr.LowerCaseFilterFactory" />
  </analyzer>
</fieldType>

3) Finally, in the schema.xml file I added the < dynamicField > "_csm" specified in the :

<dynamicField name="*_csm" type="string" indexed="true" stored="true" multiValued="true" />

When I rebuilt the index, my field was correctly tokenized by Solr.