5

Here is the situation: I have three items - Categories - Corporate, Financial and Product which I am using to categorize other items. They can be selected from a Droplink field (so Sitecore holds ID in this field).

Items which are being categorized had been imported and had been assigned one of the three IDs, corresponding to my categories (Corporate, Financial and Product).

In order to correctly index values of those fields in Lucene I have implemented simple FieldReader which converts the category field to LookupField and then reads either one of the fileds of the category item or it's name (so it does not index the ID which is stored in the field but the name of the category).

This worked like a charm when I have been manually adding my items and have been assigning categories by hand. However when I imported them, strange things started to happen:

Instead of three possible categories I have 5: Corporate, Corpora, Financial, Product, Produc.

The first thing I have checked is my custom field reader but it is always reading one of the three categories, because I only have three items representing categories.

When I attach debugger to LuceneDocumentBuilder and add conditial breakpoint which should react on "Corpora" being added to the index it never breaks there. Yet in the index there is this strange creature.

Now the things get even more interesting when I change the name of the category, to let's say 'Historical' - this one is indexed only once in its correct form. However when I introduce a typo (eg. Croporate) then it follows this strange behavior and I got "Cropora" and "Croporate" in the index.

I have removed the indexes couple of times and this didn't help. The name of the field is pretty unique so the strange value should not come from elswhere besides it disappears when I change from Corporate to something entirely different.

Does any of you has any ideas where I should look for the source of this, pretty strange, issue?

  • 1
    This is related to the KeywordAnalyzer Lucene uses for your fields. Please share your configuration snippets for the relevant indexed fields. – Mark Cassidy May 22 '17 at 11:12
  • 1
    Essentially; you are likely using the StandardAnalyzer instead of the KeywordAnalyzer. StandardAnalyzer has word stemming; which can reduce "Corporate" to "Corpora". stackoverflow.com/questions/5483903/… – Mark Cassidy May 22 '17 at 11:18
  • Ok I think I need to check the search configuration as I am using SXA and rely on default analyzer. – Szymon Kuzniak May 22 '17 at 11:29
  • Ok, found and fix. Please correct me if I did somethign stupid - since this is a category it does not have to be tokenized so I changed the configuration of the field to UNTOKENIZED and this solved a problem. Also please make this an answer so I can upvote and accept - this also really helps others to see there is an answer. – Szymon Kuzniak May 22 '17 at 11:41
4

Lucene Analyzers and Tokenizers

Essentially, Lucene is preparing your content to be the target of a general "free text search". In doing so, Lucene will break down the terms it parses in your content - a process known as word-stemming. It also filters out stop words and other things you would normally expect.

It then Tokenizes your content, breaking it down into the component parts you show us.

There are lots of resources available on the web, explaining this. Here is one example:

The solution

Stop Lucene from Analyzing and Tokenizing the fields you do not want this behaviour on. Usually, field definitions for Lucene can be TOKENIZED and UNTOKENIZED (turns Tokenization on and off). Also, the default for the Lucene StandardAnalyzer is TOKENIZED. You can switch your Analyzer to KeywordAnalyzer which will treat your content differently.

Additional resources

  • 1
    This basically solves the issue. Thanks a lot. – Szymon Kuzniak May 22 '17 at 12:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.