Secured document I have permission to not showing in Coveo search

Question

This is using Sitecore 8.1 Update 1, Coveo on-premises version 7 (8388.7) free edition, and Coveo for Sitecore version 4.0.222.0.

I have a document that's been secured such that everyone is denied access to it in inheritance, but then certain groups are granted read permissions, so that only those in the specified groups can see the content. Looking at the Effective Combined Permissions list for my item in the index browser, if I search for my account I see that it is marked as Allowed, so I'd expect to see the content in my search results. However, when I perform a search, the content isn't visible in the results.

The only advanced expression I have in the query is that it needs to have a layout, be part of a particular site, or it needs to be in a particular path. In this case, the path qualifier comes in as well as having a layout. So nothing is immediately jumping out as to why my content wouldn't appear on the site. I can browse to the content just fine. In my Sitecore logs, I do find the following error multiple times, for user accounts I know to exist...this is a good example. I'm not sure if this error could be a reason the security isn't being honored correctly.

10804 00:00:51 ERROR An error occurred while calling method "GetUserMembers".
Exception: Coveo.Connectors.Sitecore2.SitecoreWebServiceExceptions.SitecoreWebServiceUserNotFoundException
Message: The user "xyz\anonymous" does not exist.
Source: Coveo.Connectors.Sitecore2.SitecoreWebService
   at Coveo.Connectors.Sitecore2.SitecoreWebService.Wrapper.BaseSitecoreWrapper.GetUserMembers(String p_UserName)
   at Coveo.Connectors.Sitecore2.SitecoreWebService.SitecoreWebService.<>c__DisplayClass1d.<GetUserMembers>b__1c()
   at Coveo.Connectors.Sitecore2.SitecoreWebService.SitecoreWebService.TryCatchWrapper[T](Func`1 p_Action, String p_MethodName)

The Coveo diagnostic page shows all green, except for the Coveo Search Web Service, which presents this error. I don't know if there's a possible correlation here.

System.ArgumentException: Invalid JSON primitive: .
   at System.Web.Script.Serialization.JavaScriptObjectDeserializer.DeserializePrimitiveObject()
   at System.Web.Script.Serialization.JavaScriptObjectDeserializer.DeserializeInternal(Int32 depth)
   at System.Web.Script.Serialization.JavaScriptObjectDeserializer.BasicDeserialize(String input, Int32 depthLimit, JavaScriptSerializer serializer)
   at System.Web.Script.Serialization.JavaScriptSerializer.Deserialize(JavaScriptSerializer serializer, String input, Type type, Int32 depthLimit)
   at System.Web.Script.Serialization.JavaScriptSerializer.Deserialize[T](String input)
   at Coveo.Framework.Utils.JsonSerializer.DeserializeObjectFromString[T](String p_String)
   at Coveo.SearchServiceProvider.Rest.ResponseParser.ParseQueryResults(String p_JsonResponse)
   at Coveo.SearchServiceProvider.Rest.ClientSessionWrapper.ExecuteQuery(QueryParams p_QueryParams)
   at Coveo.SearchProvider.Applications.OnPremiseStateVerifier.VerifySearchService()
   at Coveo.SearchProvider.Applications.OnPremiseStateVerifier.<>c__DisplayClass14.<GetSearchServiceState>b__13()
   at Coveo.SearchProvider.Applications.BaseVerifier.VerifyComponent(Func`1 p_VerifyMethod, String p_ComponentName)

Answer 1

The answer turned out to be in the way Sitecore manages users, even if you're using an external membership database. Let's say you have a standard membership and roles database set up, with Sitecore configured to connect a custom domain and its users/roles to it. When you create a user, you'll find the record in the aspnet_Users/aspnet_Membership tables of your external database.

What Sitecore also does, however, is to create a record in the aspnet_Users table of the Core database. The user ID doesn't match (or have to), just the username is present with the domain (even if, in your external membership configuration, you specified to not store the domain with the username). This "marker" record is the key; without it, Coveo will throw those errors about not finding a user. (Also without it, setting security on a Sitecore item with an individual user won't work properly, though role-based security will.)

One thing I was doing was using a membership/role provider specifying the custom provider name. This is NOT necessary; use the standard provider calls and Sitecore will organize things correctly in the background. Since we migrated this external database from another application, we had to manually insert the "marker" records into the Core database. After doing that and clearing up the Coveo security cache, everything was good to go.