5

In our Sitecore site we have a few 'English' languages:

  • en
  • en-GB
  • en-US

We wanted to do a search using the Lucene index (a custom one, but that doesn't actually matter) and included the language in it to get only results for the current language. But when we search on "en" we also get results for the other "en" based languages. When we search on "en-gb" we get nothing.

We tried Luke and got the same results. So I assumed the field would be tokenized and indeed, in the default Lucene config I found:

<field fieldName="_language" storageType="YES" indexType="TOKENIZED" vectorType="NO" boost="1f" type="System.String" settingType="Sitecore.ContentSearch.LuceneProvider.LuceneSearchFieldConfiguration, Sitecore.ContentSearch.LuceneProvider">
  <analyzer type="Sitecore.ContentSearch.LuceneProvider.Analyzers.LowerCaseKeywordAnalyzer, Sitecore.ContentSearch.LuceneProvider" />
</field>

This explained the behavior, but now I am wondering if we can search on language (when having multiple 'English' languages) without adding a custom field for this?

  • Why o why would they make this field tokenized? I would rebel and change it to UNTOKENIZED if I were you. See if anything breaks. – Mark Cassidy Oct 3 '16 at 8:59
  • That was the first idea, but indeed wondering if that would break stuff as this is the standard config... I was hoping someone would have seen and fixed this before. – Gatogordo Oct 3 '16 at 9:02
3

Apparently the fact that we were using a custom index was an important factor after all. Sitecore Support gave us a solution:

It seems that if you have a custom index with its own configuration the language is stored in the index in a short format. We had to include the _language field as defined in the standard config in the fieldmap of our custom index. So we add this to the custom fieldmap:

<field fieldName="_language" storageType="YES" indexType="TOKENIZED" vectorType="NO" boost="1f" type="System.String" settingType="Sitecore.ContentSearch.LuceneProvider.LuceneSearchFieldConfiguration, Sitecore.ContentSearch.LuceneProvider">
  <analyzer type="Sitecore.ContentSearch.LuceneProvider.Analyzers.LowerCaseKeywordAnalyzer, Sitecore.ContentSearch.LuceneProvider" />
</field>

It seemed weird that this fixes our issue, but it does. Somehow I expected the standard Sitecore fields in the custom index to come from the default configuration but apparently they do not.

I also checked the search log files (should have done that sooner) and now the queries show full languages instead of short ones as before. And we get the correct results.

| improve this answer | |
  • 2
    I sometimes suspect; someone somewhere does not understand Lucene :P Short ID; short format language names... Seems to me like jumping through hoops, as opposed to just not tokenizing content that should not be tokenized. /rant :P – Mark Cassidy Oct 3 '16 at 15:40
  • Just a note: While the field is TOKENIZED it uses Sitecore's LowerCaseKeywordAnalyzer which just like untokenized but case-insensitive (while true untokenized is case-sensitive). – Søren Kruse Feb 3 '17 at 8:33
2

Well the obvious non-breaking alternative would be, to just add a sister field definition.

<field fieldName="_searchlanguage" storageType="YES" indexType="UNTOKENIZED" vectorType="NO" boost="1f" type="System.String" settingType="Sitecore.ContentSearch.LuceneProvider.LuceneSearchFieldConfiguration, Sitecore.ContentSearch.LuceneProvider">
  <analyzer type="Sitecore.ContentSearch.LuceneProvider.Analyzers.LowerCaseKeywordAnalyzer, Sitecore.ContentSearch.LuceneProvider" />
</field>

But as mentioned in my comment above; I don't see what use a TOKENIZED language field could be. Not to Sitecore, or anyone for that matter.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.