Running asynchronous tasks to update user profile

Question

I am looking for the best way to synchronize profile data from an external system.

We authenticate users using an external authentication service. We use the Sitecore virtual user mechanism to log into Sitecore. Once a user is logged in we need to call a back-office service to get the user profile information. We intend to sync this information with the Sitecore Contact (in XDb).

I want the synchronization of profile data to run as an asynchronous process because the backend service is slow and takes some time to respond. Also potentially we can have a lot of people log in at the same time.

We do not need the profile data immediately (but as soon as possible).

The obvious way to fire an asynchronous task would be to use a Sitecore Job. This would mean that I would fire off a job to synchronize profile data every time a user logs in. I am afraid that this will have a negative impact on performance when lots of users log in and it is not clear to me how these Jobs are prioritized.

I think I am looking for some sort of queuing system (more or less like the Sitecore publishing queue) to handle these tasks. The event queue does also not seem to meet my requirements since this is actually more a messaging than a queuing mechanism.

What would be the best solution to handle this requirement? Preferably keeping as close to Sitecore as possible.

Answer 1

Sitecore jobs is the best option in such case, but you can use them in different ways.

Best approach for using them may vary by business requirements, like frequency of user logins and the time frame in which you want to update profiles.

Below are some approaches -

1. Once the user is logged in, create a job to update that user's profile data. In case there are multiple users are getting logged in, their jobs will get queued.

   Note: Sitecore allows maximum 20 Worker Threads to be created at a time. So, when there are 20+ jobs, it may get stuck or degrade performance. In such case, you can set more number of a worker threads by adding below setting to config, but you never know how many users can get logged in at a time.

<setting name="MaxWorkerThreads" value="100"/>

2. As an alternate, You can also create jobs in separate threads by setting ExecuteInManagedThreadPool property of JobOptions to false. This will give freedom from the predefined limit of worker threads.

3. Another better alternative is (depends on how much time you need to update profile data) - When user logs in, add its details to a static C# Queue. Run a scheduler every few minutes to update their profile data. This will create only one job for multiple users.

Answer 2

You mentioned that your backoffice service was slow to respond so I am assuming that the reason for your change is because your user login is slow. Another way you can approach this is to cache the data closer to the application for faster response time. You can create a Mongo collection (or in some other data store) and update it regularly with a job. Then update your Sitecore website to query that source instead for the data, avoiding the delays your backoffice is introducing.

The advantage here is that you can take all the scheduling out of Sitecore and put it into a separate application. You can also wrap it with a service layer so you can keep a similar interface to what you have now, just point it at a different data store.

Answer 3

You could spawn a new thread to do your long running task.

public static void Getprofiledataasync(string email){
   Thread t = new Thread ( () => GetProfileData(email) );
   t.Start();
}

private static void GetProfileData(string email){
   ...
}

See this link for more info. http://www.albahari.com/threading/

Answer 4

Does your data from the back office change a lot?
Is it necessary that the data is always 100% up to date right away?

I would recommend to write a complete import (create / update) of the data to contacts in xDB. Which is something that can be scheduled to be done every weekend for example, when the load on the site is low.
The advantage is that as soon as you log in a user (and identify the contact), you'll already have its xDB data available.

You can still trigger an extra update when the user logs in. A Sitecore Job like Yogesh suggests is also my preferred way if the service is slow.