Send email with Segmented List - Skipped during message delivery

Question

I have a Sitecore v9.0u1 solution that runs in the Azure.The Sitecore instance and the EXM, Marketing Automation and IndexWorker are all hosted on the same web server for this POC project, and the databases are using an Azure SQL database, and search uses Azure Search.

For testing, I am using Gmail as my SMTP for sending EXM campaigns. The email sending part using contact list is working fine, however the segment list doesn't seem to work properly.

The process of creating email campaign using EXM as follow:

1. Create email campaign
2. Choose the recipients (segmented list) as shown below
3. Populate the message body
4. Review - sending email is working
5. Send Message

At this point, I noticed that the 'Total recipients' changed to '0' from '2'

So I jumped into the EXM.Master database which shows that the recepients were truly skipped.

SELECT * FROM dbo.campaigns WHERE [SkippedRecipients] >= 0

In the exm log file, I have enabled the 'exm.debug' to show verbose logging.

9540 11:30:46 DEBUG RecipientManager.AreListsReady for message {3EA0305A-B0A3-470F-900D-7F3C3C8764F9} skipped checking list id b461c766-c59c-4416-a040-434a5c3d2ee8 as it is Segmented 14084 11:30:49 INFO Registered email: [email protected] 14084 11:30:49 DEBUG Job Sending message (3EA0305AB0A3470F900D7F3C3C8764F9) not running. Starting. ManagedPoolThread #0 11:30:49 INFO SendingManager: SendCore() begin ManagedPoolThread #0 11:30:49 INFO Dispatch Message (Test5): Started Message Id:
{3EA0305A-B0A3-470F-900D-7F3C3C8764F9} Message Path:
/sitecore/content/Email/Messages/2018/03/11T092425/Test5 Included Recipient Lists: Test - Segmentation from XDB Contacts Excluded Recipient Lists:

ManagedPoolThread #0 11:30:53 INFO Queueing 2 contacts on 1 threads. 0 ABn contacts. 14280 11:30:53 INFO Queueing 2 contacts with 0 ABn contacts on thread 0 14280 11:30:53 DEBUG 'Test5': 2 recipients were added to the queue on thread 0 ManagedPoolThread #0 11:30:53 INFO 'Test5': 2 recipients with 0 ABn recipients were added to the queue.

MessageTaskRunner worker thread 20 11:30:53 DEBUG [DefaultEmailAddressHistoryManager] Did not submit message. No entries. MessageTaskRunner worker thread 20 11:30:53 INFO E-mail dispatch worker thread 'MessageTaskRunner worker thread 20' did not find any active tasks and exits. MessageTaskRunner worker thread 12 11:30:53 DEBUG [DefaultSentMessageManager] Did not submit message. Message id: 3ea0305a-b0a3-470f-900d-7f3c3c8764f9. Instance id: 3ea0305a-b0a3-470f-900d-7f3c3c8764f9. Emulation mode: False. Exclude from reports: False: sentContactEntries: 0 MessageTaskRunner worker thread 12 11:30:53 DEBUG [DefaultEmailAddressHistoryManager] Did not submit message. No entries. MessageTaskRunner worker thread 12 11:30:53 INFO E-mail dispatch worker thread 'MessageTaskRunner worker thread 12' did not find any active tasks and exits. MessageTaskRunner worker thread 11 11:30:53 INFO Recipient Alias PII removed skipped due to the recipient has been added to OptOut list during the sending process. MessageTaskRunner worker thread 11 11:30:53 INFO MarkMessageSent: 00:00:00.0115607 MessageTaskRunner worker thread 11 11:30:53 INFO Detailed time statistics for 'Test5' (Alias PII removed) Process the message: 00:00:00.0240374 Generate the message: 00:00:00 Get page (render page, correct html): 00:00:00 Collect files (embedded images) in memory: 00:00:00 Generate MIME: 00:00:00 Insert files (embedded images) to MIME: 00:00:00 Personalize (replace $tokens$, insert campaign event ID): 00:00:00 Send the message: 03/12/2018 03:30:53

MessageTaskRunner worker thread 11 11:30:53 INFO Recipient Alias PII removed skipped due to the recipient has been added to OptOut list during the sending process. MessageTaskRunner worker thread 11 11:30:53 INFO MarkMessageSent: 00:00:00.0059803 MessageTaskRunner worker thread 11 11:30:53 INFO Detailed time statistics for 'Test5' (Alias PII removed) Process the message: 00:00:00.0109894 Generate the message: 00:00:00 Get page (render page, correct html): 00:00:00 Collect files (embedded images) in memory: 00:00:00
Generate MIME: 00:00:00 Insert files (embedded images) to MIME: 00:00:00 Personalize (replace $tokens$, insert campaign event ID): 00:00:00 Send the message: 03/12/2018 03:30:53

MessageTaskRunner worker thread 11 11:30:53 DEBUG [DefaultSentMessageManager] Did not submit message. Message id: 3ea0305a-b0a3-470f-900d-7f3c3c8764f9. Instance id: 3ea0305a-b0a3-470f-900d-7f3c3c8764f9. Emulation mode: False. Exclude from reports: False: sentContactEntries: 0 MessageTaskRunner worker thread 11 11:30:53 DEBUG [DefaultEmailAddressHistoryManager] Did not submit message. No entries. MessageTaskRunner worker thread 11 11:30:53 INFO E-mail dispatch worker thread 'MessageTaskRunner worker thread 11' did not find any active tasks and exits. ManagedPoolThread

0 11:30:55 INFO Message 'Test5': Recipients processed: 2. Recipients skipped: 2. Recipients failed: 0. Emails/s: 2.

Average time to load batch from queue: 2.2283 ms Average time to load contact batch: 67.6949 ms Dispatch batch size: 100 Generate: min: 00:00:00; avg: 00:00:00; max: 00:00:00; total: 00:00:00. Send: min: 00:00:00.0109894; avg: 00:00:00.0175134; max: 00:00:00.0240374; total: 00:00:00.0350268. Process: min: 00:00:00.0109894; avg: 00:00:00.0175134; max: 00:00:00.0240374; total: 00:00:00.0350268. ManagedPoolThread #0 11:30:55 INFO Time to send: 2222 ms ManagedPoolThread #0 11:30:56 INFO Dispatch Message (Test5): Finished Result: FINISHED Message Id:
{3EA0305A-B0A3-470F-900D-7F3C3C8764F9} Message Path:
/sitecore/content/Email/Messages/2018/03/11T092425/Test5 Recipients Processed: 0 Message Total Sent Recipients: 0 Message Total Failed Recipients: 0 Message Total Skipped Recipients (Enable log DEBUG level to see why): 2 Included Recipient Lists: Test - Segmentation from XDB Contacts Excluded Recipient Lists:

1. Contact List from CSV file (2 recipients) - Working and able to send an email to the recipients.
2. Segmented List from Contact List which contains CSV file (2 recipients) - Working and able to send an email to the recipients

3. Segmented List from XDB Contact (2 recipients) based on the segmented rule - Not working, the recipients were skipped.

Moreover, I don't see any errors in my marketing automation log files and my index worker log files.

[Update-1] 03-14-2018

I've decompiled the DispatchTask which is responsible for the sending of the email.

Sitecore.EmailExperience.ContentManagement.config

<dispatchTask type="PROJECTABC.Feature.EmailCampaign.CM.Dispatch.DispatchTask1, PROJECTABC.Feature.EmailCampaign">
    <param ref="exm/dispatchFailedTaskPool"/>
    <param desc="recipientValidator" ref="exm/recipientValidator" />
    <param desc="contactService" ref="exm/contactService" />
    <param desc="dataProvider" ref="exm/dataProvider" />
    <param desc="itemUtil" ref="exm/itemUtil" />
    <param desc="eventDataService" ref="exm/eventDataService" />
    <param desc="dispatchManager" ref="exm/dispatchManager" />
    <param desc="emailAddressHistoryManager" ref="exm/emailAddressHistoryManager" />
    <param desc="recipientManagerFactory" ref="exm/recipientManagerFactory" />
    <param desc="sentHistoryManager" ref="exm/sentHistoryManager" />
  </dispatchTask>

DispatchTask1.cs

protected override ProgressFeedback OnSendToNextRecipient()  {
.
.
.
}

After some debugging, I noticed that it is being skipped because the MessageType is Regular .. which seems not right.

bool flag2 = this.Message.MessageType == MessageType.Regular ? this._recipientValidator1.IsSubscribed(contact, this._recipientManager1) : !this.Message.ManagerRoot.GlobalSubscription.IsInDefaultExcludeCollection(contact);

[Update-2] 03-14-2018 This has been confirmed a sitecore bug. See the solution from Steve.

In the log file, I can see that the contacts were processed and sent.

MessageTaskRunner worker thread 11 23:49:14 INFO  Message dispatch has started. Subject: 'The dispatch process is completed'. Recipient:  'PII removed'.
ManagedPoolThread #0 23:49:14 INFO  Dispatch Message (Test1): Finished
  Result:          FINISHED
  Message Id:      {8ED5799F-8010-4EE6-8206-8388E6DA91FC}
  Message Path:    /sitecore/content/Email/Messages/2018/03/13T102823/Test1
  Recipients Processed:    2
  Message Total Sent Recipients:    2
  Message Total Failed Recipients:  0
  Message Total Skipped Recipients (Enable log DEBUG level to see why):  0
  Included Recipient Lists: Test - Segmentation from XDB Contacts
  Excluded Recipient Lists: 

ManagedPoolThread #0 23:49:14 INFO  SendingManager: SendCore() end
MessageTaskRunner worker thread 11 23:49:15 INFO  Message dispatch has been completed. Subject: 'The dispatch process is completed'. Recipient: 'PII removed'.
MessageTaskRunner worker thread 11 23:49:15 INFO  Detailed time statistics for 'Dispatch Completed' ()
Process the message: 00:00:00.8906401
 Generate the message: 00:00:00
   Load user: 00:00:00
   Get page (render page, correct html): 00:00:00
   Collect files (embedded images) in memory: 00:00:00
   Generate MIME: 00:00:00
     Insert files (embedded images) to MIME: 00:00:00
     Personalize (replace $tokens$, insert campaign event ID): 00:00:00
 Send the message: 00:00:00.8281045

MessageTaskRunner worker thread 11 23:49:15 INFO  E-mail dispatch worker thread 'MessageTaskRunner worker thread 11' did not find any active tasks and exits.
ManagedPoolThread #3 23:49:15 INFO  Test Dispatch Message (Dispatch Completed): Finished
  Result:          FINISHED
  Message Id:      {17053B00-9313-4922-99BB-5FC6674ECFEA}
  Message Path:    /sitecore/content/Email/Messages/Service Messages/Status Messages/Dispatch Completed
  Recipients Processed:    0

Answer 1

Thanks for the comprehensive bug report. Sitecore have published a hotfix for Sitecore 9.0 Update 1 which fixes this issue.

The decompiled code is hard to read but the issue stems from flag2 (as it appears here) being too strict. It has been adjusted to now simply check the Global Opt-out list for both Regular and Automated messages.

Please ensure this update is installed on all instances, not just Content Management, if you are operating a scaled setup.