How to query Solr with spaces and slashes in the filters?

Question

I have a Sitecore 8.1 instance using Solr 4.10. I have a custom field in my Solr index called read_roles. It is a list of Sitecore roles that can read a given item. Here is an example of what that looks like in the index when viewed through the Solr UI.

"read_roles_sm": [
          "extranet\\DistribA",
          "extranet\\Anonymous Role",
          "extranet\\Approved Role" ]

My first question is this: Are these values really stored with two slashes like that? Or is a slash a special character or something and the Solr UI is adding that in only when it is being displayed on the web page?

I have code that adds a security filter to my query that looks like this:

public static IQueryable<T> ApplySecurityFilter<T>(this IQueryable<T> query) where T : SearchResultItem {

    var userRoles = Sitecore.Context.User.Roles.Select(r => r.Name);

    var readPredicate = PredicateBuilder.False<T>();
    readPredicate = userRoles.Aggregate(readPredicate, (current, role) => current.Or(i => i["read_roles"].Equals(role)));

    var denyPredicate = PredicateBuilder.True<T>();
    denyPredicate = userRoles.Aggregate(denyPredicate, (current, role) => current.And(i => !i["denied_roles"].Equals(role)));

    if (readPredicate.Body.NodeType != System.Linq.Expressions.ExpressionType.Constant) {
        query = query.Filter(readPredicate);
    }
    if (readPredicate.Body.NodeType != System.Linq.Expressions.ExpressionType.Constant) {
        query = query.Filter(denyPredicate);
    }
    return query;
}

The idea here is give me all results where at least one of the user's roles is in the read_roles field and none of the user's roles are in the denied_roles field.

I seem to be running in to all kinds of problems with slashes and spaces and I don't really understand what is going on here and I don't know how to troubleshoot it. I believe this code works fine as long as the user does not belong to any roles with a space in them.

Here is an example. If I am not logged in to the site (anonymous user which belongs to the extranet\Anonymous Role) and I try a search it works properly. I get the results that I would expect. And I see this in the Sitecore Search log file as part of the query that was sent to Solr:

&fq=(((-denied_roles_sm:("extranet\\Anonymous Role") AND read_roles_sm:("extranet\\Anonymous Role"))

However if I log in as a user that is part of the extranet\DistribA role I do not get the results I would expect and I see this in the Sitecore Search log file:

&fq=(((-denied_roles_sm:(extranet\DistribA) AND read_roles_sm:(extranet\DistribA))

What I have noticed is that if the role has a space in it then it seems to put quotes around it and also adds a second slash. But if the role name does not have a space in it then it doesn't do that. I feel like I am not understanding something fundamental about how Solr handles things like spaces and slashes in queries. Any help is greatly appreciated.

Answer 1

About spaces and quotes

If there is no space, then field:foo treats foo as a string value. You could write field:"foo" but you don't have to, since the search term doesn't contain a space.

If there is a space included, then quotes are required around the value, otherwise the query won't be parsed correctly: field:"foo bar"

About backslashes

Backslashes are not stored like this: \\. It's just the escaping syntax that is used both in Solr queries and in the JSON representation in the Solr UI.

A backslash should actually always be escaped - see this documentation section:

http://lucene.apache.org/core/2_9_4/queryparsersyntax.html#Escaping%20Special%20Characters

I think that in case of extranet\DistribA, Sitecore's query generator has a defect that does not escape it. It's probably related to the fact that there are no quotes around the search term.

Suggested fix

I suggest that you don't store backslashes in the index so as to overcome this defect. Sanitize your role names to remove the backslash—both when storing the custom field and when querying the index.

Answer 2

I think what you may need is a new field type. The _sm field is a string field in the Solr schema.xml, specifically a multivalued string field. There's no tokenizer specified, so I believe that means it uses the default tokenizer. What you want is the KeywordTokenizerFactory. The KeywordTokenizerFactory treats the entire value as a literal, so it includes spaces when matching. It's also case sensitive, but you can add a LowerCaseFilterFactory if that's an issue.

Try this in the schema.xml,

<fields>
    ...    
        <dynamicField name="*_sm" type="string" indexed="true" stored="true" multiValued="true" />
        <dynamicField name="*_smnt" type="string_untokenized" indexed="true" stored="true" multiValued="true" />
    ...
</fields>
...
<types>
    ...
    <fieldType name="string" class="solr.StrField" sortMissingLast="true" />
    <fieldType name="string_untokenized" class="solr.TextField" sortMissingLast="true" >
        <analyzer>
            <tokenizer class="solr.KeywordTokenizerFactory"/>
            <filter class="solr.LowerCaseFilterFactory"/>
        </analyzer>
    </fieldType>
    ...
<types>

Then in your Sitecore Solr config you need to add this field type, like so

<fieldMap type="Sitecore.ContentSearch.SolrProvider.SolrFieldMap, Sitecore.ContentSearch.SolrProvider">
    <!-- This element must be first -->
    <typeMatches hint="raw:AddTypeMatch">
        <typeMatch typeName="string_multi_untokenized"  type="System.String"  fieldNameFormat="{0}_smnt"  settingType="Sitecore.ContentSearch.SolrProvider.SolrSearchFieldConfiguration, Sitecore.ContentSearch.SolrProvider" />
    </typeMatches>
    <!-- This allows you to map a field name in Sitecore to the index and store it in the appropriate way -->
    <fieldNames hint="raw:AddFieldByFieldName">
        <field fieldName="read_roles"     returnType="string_multi_untokenized" />
    </fieldNames>
</fieldMap>

Answer 3

Have a look at the sitecore helper Sitecore.ContentSearch.LuceneProvider.LuceneSearchContext which is calling lucene's Lucene.Net.QueryParsers.QueryParser. It escapes special characters. Thank you @anicho for pointing it out on slack