Engagement Plans: Why is a user not added to an engagement plan until the session of a user is timed out?

Question

In short my question is: Why is a user not added to an engagement plan until the session of a user is timed out?

My setup: I am running Sitecore 8.1.3 and I have created a simple engagement plan, which simply sends a confirmation mail to a user stating that he has subscribed to a newsletter. It consists of 3 elements:

1. Initial state with a timeout of 1 minute
2. Condition which is always evaluated to true
3. Action which sends a campaign (EXM) mail to the user

I add the user to the engagement via a rule action on a goal.

The waiting game: When the user submits to a newsletter I trigger the goal and from the logs, I can see that the user is added to the engagement plan, however I can't see the user in the Engagement Plan supervisor and no mails are send. Only when the session of the user times out (20 minutes after the last page view), the user is added to the engagement plan, which i can verify by looking at the xDB and using the EA supervisor. Only after I can see that the user is indeed added to the engagement plan, he is process through the plan.

Have I misunderstood something, or are engagement plans not suitable for sending notifications to users immediately? I know that i could simply add an action to the goal that sends an email, but my customer plans to do more elaborate things as part of the sign-up process.

Edit: I know that Sitecore provides a module called "Live session agent", but I am unsure if that would solve my questions, based on the documentation that Sitecore provided (https://dev.sitecore.net/Downloads/Engagement_Automation_Live_Session_Agent.aspx)

Answer 1

This is because the user is not flushed to MongoDB before SessionEnd. There are various reasons for this¹, one being that you want to minimize the writes to MongoDB.

If you need to do something immediately you can manually flush the user session, but keep in mind that it can have side effects, so consider if it's absolutely required.

You can abandon the session like this:

Sitecore.Analytics.Tracker.Current.EndTracking();
HttpContext.Current.Session.Abandon();

¹ There's a bit more to it than just performance. Once you have a user session that contact is locked (to that specific web cluster) for the duration of the session. This mean other server roles will not be able to modify the contact during this period. Consider this scenario: You have a job running on your CM server which synchronises contact information from your CRM system. At the same time the user is browsing the website (on the CD server) and updates her contact info. Sitecore knows how to merge your changes upon flushing the user session, but you have to be aware of this yourself if you create a job which needs to modify the user.

Answer 2

This is because xDB data is not committed until after the user's session ends.

This link contains a little more information on why that is: https://doc.sitecore.net/sitecore_experience_platform/setting_up__maintaining/xdb/session_state/session_state

Session state is a way of identifying contacts on your website by assigning them a unique session ID and by using cookies. Every time a contact makes a request, such as triggering a campaign or a goal this information is held in session state until the end of session.

This means that fewer server trips are needed to write data back and forth to the collection database. Instead, xDB makes one trip at the start of a session to identify the contact and a second at the end to save all the session data back to the collection database.

The xDB data is committed in the postSessionEnd pipeline if I'm not mistaken. There is also the sessionEnd pipeline which handles some analytics stuff among other things and is run before the postSessionEnd pipeline (obviously).

<sessionEnd>
  <!-- ... -->
  <processor type="Sitecore.Analytics.Pipelines.SessionEnd.InitializeTracker, Sitecore.Analytics" patch:source="Sitecore.Analytics.Tracking.config"/>
  <processor type="Sitecore.Analytics.Pipelines.SessionEnd.RaiseVisitEnd,Sitecore.Analytics" patch:source="Sitecore.Analytics.config"/>
</sessionEnd>
<postSessionEnd>
  <processor type="Sitecore.Analytics.Pipelines.PostSessionEnd.CommitSession, Sitecore.Analytics" patch:source="Sitecore.Analytics.Tracking.Database.config"/>
</postSessionEnd>

I don't know enough about engagement plans to offer any advice on wether or not it is suited for sending notifications immediately.

Answer 3

In previous incarnations, when the Experience Platform was the DMS it was all very chatty to the SQL server; for busy sites it really hammered it and we saw a few customers just turn it off.

It's matured now and data is flushed through to the xDB in one hit after the users session times ends. Which might be 20 mins, or could be longer if you have a very engaging site!

So the short answer is no, you can't rely on it for instant notifications and will probably need a separate solution.

Alternatively in production, I suppose it's worth considering if 20 mins for confirmation of a sign-up is actually a problem; It will be if they need to validate themselves, it won't be if it's just for reference.

For development purposes as https://sitecore.stackexchange.com/users/121/jacob-nielsen mentioned you can flush the session to push the user onto the next step in the journey. Personally I would use this for development and test only.

References

https://doc.sitecore.net/sitecore_experience_platform/setting_up__maintaining/xdb/platform/architecture_overview