Solr spellchecker inconsistent behavior when first character is misspelled

Question

I'm using Solr 7.2.1, I'm implementing did you mean functionality using Solr spellchecker, It's working fine except for some cases that I don't understand its behavior, like in my dictionary I have a word 'service', if I wrote 'gervice', 'service' is returned as a corrected word which is right.

But if I wrote 'gervic' it doesn't return any thing although the maxEdits is 2.

This case happens when the first character is misspelled, the max edit step number becomes 1, it doesn't correct any other characters.

Here are my configurations, I have small dictionary like 3000 words, I tried commenting thresholdTokenFrequency but it didn't work also.

 <!-- a spellchecker built from a field of the main index -->
    <lst name="spellchecker">
      <str name="name">default</str>
      <str name="field">title_t</str>
      <str name="classname">solr.DirectSolrSpellChecker</str>
      <!-- the spellcheck distance measure used, the default is the internal levenshtein -->
      <str name="distanceMeasure">internal</str>
      <!-- minimum accuracy needed to be considered a valid spellcheck suggestion -->
      <float name="accuracy">0.5</float>
      <!-- the maximum #edits we consider when enumerating terms: can be 1 or 2 -->
      <int name="maxEdits">2</int>
      <!-- the minimum shared prefix when enumerating terms -->
      <int name="minPrefix">0</int>
      <!-- maximum number of inspections per result. -->
      <int name="maxInspections">5</int>
      <!-- minimum length of a query term to be considered for correction -->
      <int name="minQueryLength">3</int>
      <!-- maximum threshold of documents a query term can appear to be considered for correction -->
      <float name="maxQueryFrequency">0.00</float>
      <!-- uncomment this to require suggestions to occur in 1% of the documents  -->
        <float name="thresholdTokenFrequency">0.00001</float>
    </lst> 


 <requestHandler name="/spell" class="solr.SearchHandler" startup="lazy">
    <lst name="defaults">
      <!-- Solr will use suggestions from both the 'default' spellchecker
           and from the 'wordbreak' spellchecker and combine them.
           collations (re-written queries) can include a combination of
           corrections from both spellcheckers -->
      <str name="spellcheck.dictionary">default</str>
      <str name="spellcheck">on</str>
      <str name="df">title_t</str>
      <str name="spellcheck.extendedResults">true</str>
      <str name="spellcheck.count">10</str>
      <str name="spellcheck.alternativeTermCount">5</str>
      <str name="spellcheck.maxResultsForSuggest">5</str>
      <str name="spellcheck.collate">true</str>
      <str name="spellcheck.collateExtendedResults">true</str>
      <str name="spellcheck.maxCollationTries">10</str>
      <str name="spellcheck.maxCollations">5</str>
      <str name="spellcheck.build">true</str> 
    </lst>
    <arr name="last-components">
      <str>spellcheck</str>
    </arr>
  </requestHandler>

Answer 1

Spellchecking lucene algorithms are many times smarter than just "calculate edit distance between term and all possible combinations of indexed word", because of perfomance.

First character in term is very important in spellchecking, also because of performance, and I recommend you do not use minPrefix=0 setting that disabled first character equality.

For your case: gervic does not look like misspell of service and lucene algorithms understand it. You even can get different results depends on what exact character you type instead correct one.

Just an example: misspellings of word islamic

1. term = xslamic, result = islamic (distance = 1)

2. term = aslamic, result = asia (distance = 4)

It proves my words above: first character is important for spellchecking and, as you can see in my example, secord case does not process word islamic at all, because it find word asia faster, although edit distance 4 is very big in comparison with distance of word islamic.

From my practice, better implementation is a combination of Spellchecker and Suggester, that allows your users to see autocomlete in real time and prevent them from making mistakes:

SolrSuggestQuery q = term;
var options = new SuggestHandlerQueryOptions
{
    Parameters = new SuggestParameters
    {
        Count = count
    },
    Start = 0,
    Rows = count,
    Fields = new List<string> { FieldTitle}
};

var result = context.Suggest(q, options);
var titleSuggestions = result.Suggestions["mySuggester"].Suggestions.Select(x => x.Term);

var suggestions = titleSuggestions.Take(count);
if (suggestions.Any()) return suggestions;


// spell checker
var query = new SolrQuery(string.Format("{0}:{1}", FieldTitle, term));
var results = context.GetSpellCheck(query, new SpellCheckHandlerQueryOptions
{
    SpellCheck = new SpellCheckingParameters
    {
        Count = count,
    },
    Start = 0,
    Rows = count,
    Fields = new List<string> { FieldTitle}
});

if (results != null && results.SpellChecking != null && results.SpellChecking.Count > 0)
{
    var words = term.Split(new[] { ' ' }, StringSplitOptions.RemoveEmptyEntries);
    if (words.Length < 2)
    {
        return results.SpellChecking.SelectMany(x => x.Suggestions);
    }

    var correct = "";
    foreach (var w in words)
    {
        var suggest = results.SpellChecking.FirstOrDefault(x => x.Query.Equals(w));
        if (suggest != null && suggest.Suggestions.Any())
        {
            correct += suggest.Suggestions.First() + " ";
        }
        else
        {
            correct += w + " ";
        }
    }
    return new List<string> { correct.Trim() };
}

return new List<string>();

You can also try to play with different distance algorithms:

<str name="distanceMeasure">org.apache.lucene.search.spell.JaroWinklerDistance</str>
or
<str name="distanceMeasure">org.apache.lucene.search.spell.NGramDistance</str>

but for me default Levenshtein distance is more than enough.