Unable to use segmented contact list as a source for email campaign - all contacts are globally excluded?

Question

I am using EXM 3.4 on Sitecore 8.2 initial release.

If I try to import a list of contacts with CSV and send the regular campaign - it works. But if I try to create a segmented contact list in the list manager first, and then try to use it as a source - I am getting the next message: You must add at least one populated recipient list to the message.

More details: So I go to list manager and create a segmented list from all contacts, I give it a name and define the segmentation conditions. Saving it and Can see correct number of recepients and they are listed below.

Then I go to EXM and create a regular email campaign. On Recipients tab I select the previously created list:

If I go to Message tab, I can actually do the "preview fr recipient" from my list and I see the same contacts as on the list manager. But when I go to lat tab - Delivery and click "Send message" I am getting next message You must add at least one populated recipient list to the message.:

I also see the correct amount of recipients in the right sidebar "Email campaign info" But there is the same amount of customers are treated as Globally excluded:

If I check my Global Opt-Out list - it is empty and contains no recipients. My segmented list uses one custom segmentation condition which I implemented, but as I said it works fine if I look into the list on list manager and even it allows me to select the contact for the email preview feature. Any thoughts about that?

It looks like the list manager issue. If I, for example, go to my segmented list and add Global Opt-Out list which is empty as a exclude source - the list returns no contacts:

Probably the same happens in EXM when it tries to exclude global opt out's from the segmented list.

Answer 1

I have figured out this one. I am using SOLR contentSearch provider and SOLR 5.4.1 server. In my search.log file I found an error:

1880 14:10:54 ERROR Solr Error : ["org.apache.solr.search.SyntaxError: Cannot parse '((((contact.hasemail_b:(True) AND contact.isregisteredplayer_b:(True)) AND type_t:(contact)) AND (-(((contact.tags_sm:(ContactLists:{7785E04C-9481-454C-A597-9E56D7CE224C}) OR (contact.tags_sm:({7785E04C-9481-454C-A597-9E56D7CE224C}) AND contact.tags_sm:(ContactLists~0.5))) AND (contact.hasemail_b:(True) AND contact.isregisteredplayer_b:(True))) AND type_t:(contact))  *:*)) AND contact.preferredemail_t:[* TO *])': Encountered " "}" "} "" at line 1, column 169.
Was expecting one of:
    "TO" ...
    <RANGE_QUOTED> ...
    <RANGE_GOOP> ...

and this part looked suspicion to me:

contact.tags_sm:(ContactLists:{7785E04C-9481-454C-A597-9E56D7CE224C})

If I go to my SOLR server and try to execute this query - I am getting same error. This is because of {} symbols.

In SOLR I can see how contact.tags_sm field looks like:

"contact.tags_sm": [
  "ContactLists:{A94AC903-121D-43C9-94C5-04F694FE6C0C}",
  "ContactLists:{7785E04C-9481-454C-A597-9E56D7CE224C}"
],

So correct query should include string escaping and looking like this:

contact.tags_sm:("ContactLists:{7785E04C-9481-454C-A597-9E56D7CE224C}")

After few investigations I found that this query is being added in Sitecore.ListManagement.ContentSearch.ContactListContactQueryProvider<TContactData> in Sitecore.ListManagement.ContentSearch.dll class and there is a method here which does it:

protected override IQueryable<TContactData> GetContactQueryable(IProviderSearchContext searchContext)
{
    IQueryable<TContactData> source = this.IndexedContactRuleEngine != null 
        ? (IQueryable<TContactData>) this.IndexedContactRuleEngine.GetEntitiesFilteredByRules(this.IndexedContactRuleEngine.DeserializeRuleList(this.RuleListSerialization), searchContext) 
        : searchContext.GetQueryable<TContactData>();

    string tagFormat = string.Format("{0}:{1}", (object) "ContactLists", (object) this.contactListId);
    return source.Where<TContactData>((Expression<Func<TContactData, bool>>) (x => x.Tags.Contains<string>(tagFormat) || x.Tags.Contains<string>(this.contactListId) && x.Tags.Like<string[]>("ContactLists")));
}

Perhaps that will work on Lucene, but not in SOLR. So apparently, I need to change that. And here is my overload class for ContactListContactQueryProvider.

namespace Cabana.Xdb.ListManagement
{
    using System.Linq;
    using Sitecore.Analytics.Rules.SegmentBuilder;
    using Sitecore.ContentSearch;
    using Sitecore.ContentSearch.Analytics.Models;
    using Sitecore.ContentSearch.Linq;
    using Sitecore.ListManagement.ContentSearch;
    using Sitecore.ListManagement.ContentSearch.Model;

    public class ContactListContactQueryProvider<TContactData>: Sitecore.ListManagement.ContentSearch.ContactListContactQueryProvider<TContactData>
        where TContactData : ContactData
    {
        public ContactListContactQueryProvider(
                ISearchIndex index, 
                string contactListId, 
                IRuleEngine<IndexedContact, VisitorRuleContext<IndexedContact>> indexedContactRuleEngine, 
                string ruleListSerialization) 
            : base(index, contactListId, indexedContactRuleEngine, ruleListSerialization)
        {
        }

        public ContactListContactQueryProvider(
                ISearchIndex index, 
                string contactListId) 
            : base(index, contactListId)
        {
        }

        protected override IQueryable<TContactData> GetContactQueryable(IProviderSearchContext searchContext)
        {
            var source = this.IndexedContactRuleEngine != null 
                ? (IQueryable<TContactData>)this.IndexedContactRuleEngine.GetEntitiesFilteredByRules(
                    this.IndexedContactRuleEngine.DeserializeRuleList(this.RuleListSerialization), searchContext) 
                : searchContext.GetQueryable<TContactData>();

            var tagFormat = $"\"ContactLists:{this.ContactListId}\"";
            return source.Where(x => x.Tags.Contains(tagFormat) || x.Tags.Contains($"\"{this.ContactListId}\"") && x.Tags.Like("\"ContactLists\""));
        }
    }
}

Ok, now I need to patch something in configs to make it work. I figured out that there are 2 pipelines for retrieving contacts from lists:

<listManagement.getSegmentedListContacts>

<listManagement.getAssociatedContacts>

Also, the SegmentedListContactQueryProvider class depends on ContactListContactQueryProvider one (direct hardcoded usage).

So there are 3 more classes to implement:

namespace Cabana.Xdb.ListManagement
{
    using System.Linq;
    using Sitecore.Analytics.Rules.SegmentBuilder;
    using Sitecore.ContentSearch;
    using Sitecore.ContentSearch.Analytics.Models;
    using Sitecore.ListManagement.ContentSearch;
    using Sitecore.ListManagement.ContentSearch.Model;

    public class SegmentedListContactQueryProvider<TContactData> : Sitecore.ListManagement.ContentSearch.SegmentedListContactQueryProvider<TContactData> 
        where TContactData : ContactData
    {
        public SegmentedListContactQueryProvider(
                ISearchIndex index, 
                IRuleEngine<IndexedContact, VisitorRuleContext<IndexedContact>> ruleEngine, 
                string ruleListSerialization, 
                ListSource listSource) 
            : base(index, ruleEngine, ruleListSerialization, listSource)
        {
        }

        protected override IQueryable<TContactData> GetContactQueryable(IProviderSearchContext searchContext)
        {
            var queryable = Enumerable.Empty<TContactData>().AsQueryable();
            if (!this.ListSource.AllDatabase && this.ListSource.IncludedLists.Count <= 0)
            {
                return queryable;
            }

            var ruleList = this.RuleEngine.DeserializeRuleList(this.RuleListSerialization);

            if (ruleList.Count == 0 || ruleList.Rules.All(r => r.Condition == null))
            {
                return queryable;
            }

            return this.ListSource.ExcludedLists.Aggregate(!this.ListSource.AllDatabase
                ? this.ListSource.IncludedLists.Skip(1).Aggregate(this.GetContactListContacts(this.ListSource.IncludedLists.First()), (current, includedList) => current.Union(this.GetContactListContacts(includedList)))
                : (IQueryable<TContactData>)this.RuleEngine.GetEntitiesFilteredByRules(ruleList, searchContext), (current, excludedList) => current.Except(this.GetContactListContacts(excludedList)))/*.Where(c => !string.IsNullOrEmpty(c.PreferredEmail))*/;

        }
        private IQueryable<TContactData> GetContactListContacts(string listId)
        {
            return new QueryableProxy<TContactData>(new Cabana.Xdb.ListManagement.ContactListContactQueryProvider<TContactData>(this.Index, listId, this.RuleEngine, this.RuleListSerialization));
        }
    }
}

Pipeline processors:

namespace Cabana.Xdb.ListManagement.Pipelines
{
    using Sitecore.ContentSearch;
    using Sitecore.Diagnostics;
    using Sitecore.ListManagement.Configuration;
    using Sitecore.ListManagement.ContentSearch;
    using Sitecore.ListManagement.ContentSearch.Model;
    using Sitecore.ListManagement.ContentSearch.Pipelines;

    public class GetContactAssociations
    {
        private readonly ISearchIndex index;

        public GetContactAssociations()
        {
            this.index = ContentSearchManager.GetIndex(ListManagementSettings.ContactsIndexName);
        }

        public virtual void Process(GetAssociatedContactsArgs args)
        {
            Assert.ArgumentNotNull(args, nameof(args));
            args.Contacts = new QueryableProxy<ContactData>(new ListManagement.ContactListContactQueryProvider<ContactData>(this.index, args.ContactList.Id));
        }
    }
}

namespace Cabana.Xdb.ListManagement.Pipelines
{
    using System.Linq;
    using Sitecore.Analytics.Rules.SegmentBuilder;
    using Sitecore.ContentSearch;
    using Sitecore.ContentSearch.Analytics.Models;
    using Sitecore.Diagnostics;
    using Sitecore.ListManagement.Configuration;
    using Sitecore.ListManagement.ContentSearch;
    using Sitecore.ListManagement.ContentSearch.Model;
    using Sitecore.ListManagement.ContentSearch.Pipelines;

    public class GetSegmentedListContacts
    {
        private readonly ISearchIndex index;
        private readonly IRuleEngine<IndexedContact, VisitorRuleContext<IndexedContact>> ruleEngine;

        public GetSegmentedListContacts(IRuleEngine<IndexedContact, VisitorRuleContext<IndexedContact>> ruleEngine)
        {
            Assert.ArgumentNotNull(ruleEngine, nameof(ruleEngine));
            this.index = ContentSearchManager.GetIndex(ListManagementSettings.ContactsIndexName);
            this.ruleEngine = ruleEngine;
        }

        public virtual void Process(GetAssociatedContactsArgs args)
        {
            Assert.ArgumentNotNull(args, nameof(args));
            var contactList = args.ContactList as SegmentedList;

            args.Contacts = !string.IsNullOrEmpty(contactList?.Query) 
                ? new QueryableProxy<ContactData>(new Cabana.Xdb.ListManagement.SegmentedListContactQueryProvider<ContactData>(this.index, this.ruleEngine, contactList.Query, contactList.Source)) 
                : Enumerable.Empty<ContactData>().AsQueryable();
        }
    }
}

and finally patch for configuration:

<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
    <sitecore>
        <pipelines>
            <listManagement.getSegmentedListContacts>
                <processor
                    type="Cabana.Xdb.ListManagement.Pipelines.GetSegmentedListContacts, Cabana.Xdb"
                    patch:instead="*[@type='Sitecore.ListManagement.ContentSearch.Pipelines.GetAssociatedContacts.GetSegmentedListContacts, Sitecore.ListManagement.ContentSearch']">
                    <param ref="ruleEngine"/>
                </processor>
            </listManagement.getSegmentedListContacts>
            <listManagement.getAssociatedContacts>
                <processor
                    type="Cabana.Xdb.ListManagement.Pipelines.GetContactAssociations, Cabana.Xdb"
                    patch:instead="*[@type='Sitecore.ListManagement.ContentSearch.Pipelines.GetAssociatedContacts.GetContactAssociations, Sitecore.ListManagement.ContentSearch']">
                </processor>
            </listManagement.getAssociatedContacts>
        </pipelines>
    </sitecore>
</configuration>

Sounds like a Sitecore bug so probably will contact support.