2

I'm on Solr 4.10.4, Sitecore 8.1 160519.

I finally managed to convert my Lucene custom index configuration to Solr, but some indexes are not tokenized. The custom index "parsed_answers" on Lucene is defined as follows:

<field fieldName="parsed_answers" storageType="yes" indexType="tokenized" />

while on Solr is defined like so:

<field fieldName="parsed_answers" returnType="stringCollection" storageType="yes" indexType="tokenized" multiValued="true">

My custom code returns a lengthy string. Lucene breaks it down into individual words and stores each word in the index individually. Solr simply takes the entire text blurb and indexes it. This is not desirable.

Question

I have read about tokenizers in Solr:

<fieldType name="text" class="solr.TextField">
  <analyzer>
    <tokenizer class="solr.StandardTokenizerFactory"/>
  </analyzer>
</fieldType>

I could not, however, find an example that I can use in my case yet - do I have to define a new typeMatch that is tokenized, or can I tokenize an existing one?

Any help is appreciated.

2 Answers 2

1

On https://lucene.apache.org/solr/guide/6_6/field-types-included-with-solr.html#field-types-included-with-solr you can find a description of the Solr field types. (or https://archive.apache.org/dist/lucene/solr/ref-guide/apache-solr-ref-guide-4.10.pdf for Solr 4)

For "string" they mention:

String (UTF-8 encoded string or Unicode). Strings are intended for small fields and are not tokenized or analyzed in any way. They have a hard limit of slightly less than 32K.

As your stringCollection is a collection of strings, they will not be tokenized. To get that functionality, you need to use text - or in your multiple value case a textCollection.

Must admit that I never used a textCollection myself, but based on the docs that should work.

8
  • So there is no way of having a string or stringCollection automatically tokenized? Feb 15, 2019 at 9:32
  • 1
    Not exactly sure (am not a real Solr expert) but based on what I'm reading I would assume not
    – Gatogordo
    Feb 15, 2019 at 9:38
  • I don't have any textCollection under my <typeMatches>. Is this something I can freely add myself? I also could not find neither string, nor text, nor textCollection at the URL you posted. Are you sure it's the right one? Feb 15, 2019 at 11:06
  • 1
    I just checked with an old project on SC8 and Solr4 and it's not there indeed.. in a newer project with a more recent solr it is. So must be something that changed either on solr or Sitecore
    – Gatogordo
    Feb 15, 2019 at 12:54
  • I would try to add it somehow.. it's a custom index so won't break Sitecore and text is available in Solr4
    – Gatogordo
    Feb 15, 2019 at 12:59
0

As it turns out, the only way of achieving this with Solr 4.10.4 is by modifying the schema.xml file of the Solr core.

What I did was as follows:

  1. In the Sitecore custom configuration I defined a < typeMatch > called "customStringCol":

  2. I then created the same custom fieldType "customStringCol" in the schema.xml file and added a Tokenizer to it:

  3. Finally, in the schema.xml file I added the < dynamicField > "_csm" specified in the :

When I rebuilt the index, my field was correctly tokenized by Solr.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.