2

I'm on Solr 4.10.4, Sitecore 8.1 160519.

I finally managed to convert my Lucene custom index configuration to Solr, but some indexes are not tokenized. The custom index "parsed_answers" on Lucene is defined as follows:

<field fieldName="parsed_answers" storageType="yes" indexType="tokenized" />

while on Solr is defined like so:

<field fieldName="parsed_answers" returnType="stringCollection" storageType="yes" indexType="tokenized" multiValued="true">

My custom code returns a lengthy string. Lucene breaks it down into individual words and stores each word in the index individually. Solr simply takes the entire text blurb and indexes it. This is not desirable.

Question

I have read about tokenizers in Solr:

<fieldType name="text" class="solr.TextField">
  <analyzer>
    <tokenizer class="solr.StandardTokenizerFactory"/>
  </analyzer>
</fieldType>

I could not, however, find an example that I can use in my case yet - do I have to define a new typeMatch that is tokenized, or can I tokenize an existing one?

Any help is appreciated.

1

On https://lucene.apache.org/solr/guide/6_6/field-types-included-with-solr.html#field-types-included-with-solr you can find a description of the Solr field types. (or https://archive.apache.org/dist/lucene/solr/ref-guide/apache-solr-ref-guide-4.10.pdf for Solr 4)

For "string" they mention:

String (UTF-8 encoded string or Unicode). Strings are intended for small fields and are not tokenized or analyzed in any way. They have a hard limit of slightly less than 32K.

As your stringCollection is a collection of strings, they will not be tokenized. To get that functionality, you need to use text - or in your multiple value case a textCollection.

Must admit that I never used a textCollection myself, but based on the docs that should work.

  • So there is no way of having a string or stringCollection automatically tokenized? – Emanuele Ciriachi Feb 15 at 9:32
  • 1
    Not exactly sure (am not a real Solr expert) but based on what I'm reading I would assume not – Gatogordo Feb 15 at 9:38
  • I don't have any textCollection under my <typeMatches>. Is this something I can freely add myself? I also could not find neither string, nor text, nor textCollection at the URL you posted. Are you sure it's the right one? – Emanuele Ciriachi Feb 15 at 11:06
  • 1
    I just checked with an old project on SC8 and Solr4 and it's not there indeed.. in a newer project with a more recent solr it is. So must be something that changed either on solr or Sitecore – Gatogordo Feb 15 at 12:54
  • I would try to add it somehow.. it's a custom index so won't break Sitecore and text is available in Solr4 – Gatogordo Feb 15 at 12:59
1

As it turns out, the only way of achieving this with Solr 4.10.4 is by modifying the schema.xml file of the Solr core.

What I did was as follows:

1) In the Sitecore custom configuration I defined a < typeMatch > called "customStringCol":

<typeMatch typeName="customStringCol" type="System.Collections.Generic.List`1[System.String]" fieldNameFormat="{0}_csm" multiValued="true" indexed="true" settingType="Sitecore.ContentSearch.SolrProvider.SolrSearchFieldConfiguration, Sitecore.ContentSearch.SolrProvider"/>

2) I then created the same custom fieldType "customStringCol" in the schema.xml file and added a Tokenizer to it:

<fieldType name="customStringCol" class="solr.TextField" sortMissingLast="true">
  <analyzer type="index">
    <tokenizer class="solr.StandardTokenizerFactory" />
    <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" />
    <filter class="solr.LowerCaseFilterFactory" />
  </analyzer>
</fieldType>

3) Finally, in the schema.xml file I added the < dynamicField > "_csm" specified in the :

<dynamicField name="*_csm" type="string" indexed="true" stored="true" multiValued="true" />

When I rebuilt the index, my field was correctly tokenized by Solr.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.