Is there anyway to improve the speed that List Manager builds a list?

Question

In Sitecore 8.X, the List Manager is the primary tool used to import contacts and create audience lists for Sitecore. This is a dependency in EXM as well for creating message dispatches against a List Manager list.

When a list is created, it goes through a couple steps:

1. Sitecore creates the stub of the list as an item in Sitecore.
2. Sitecore then locks the list from any pending changes.
3. During the lock phase:
   • Sitecore Creates the xDB contact.
   • Sitecore adds to the sitecore_list_index the list information.
4. Once the lock phase is complete, Sitecore unlocks the list and it can be used for consumption.

The main issue at hand is that for some reason, writing to the sitecore_list_index takes an enormous amount of time. Creating a list of 2 users has thus taken 5 minutes. When I have a CSV of 90,000 that I need to upload.

A little bit about our indexing. We are using Solr Server 6 for all indexes. (Yes, we know Solr 6 isn't an approved version, but so far, we not have any issues and all tests are coming back very positive. The master index builds very fast.)

The sitecore_list_index is very small.

Question: How do I improve this experience? Am I even able to?

UPDATE 1: I don't think it's Indexes. According to the log, the index is getting processed and updated almost immediately.

ManagedPoolThread #11 16:13:33 INFO  Job started: Index_Update_IndexName=sitecore_list_index
ManagedPoolThread #9 16:13:33 INFO  Job ended: Index_Update_IndexName=sitecore_master_index (units processed: )
ManagedPoolThread #9 16:13:33 INFO  Job started: Index_Update_IndexName=sitecore_master_index
ManagedPoolThread #17 16:13:33 INFO  Job started: CONTACT_BULK_UPDATE_ListManagementImportContacts_cbaf9d6805a74a509e0413ac2fd751b7
ManagedPoolThread #11 16:13:33 INFO  Job ended: Index_Update_IndexName=sitecore_list_index (units processed: )
ManagedPoolThread #4 16:13:33 INFO  Job started: Index_Update_IndexName=sitecore_list_index
ManagedPoolThread #17 16:13:33 INFO  Job ended: CONTACT_BULK_UPDATE_ListManagementImportContacts_cbaf9d6805a74a509e0413ac2fd751b7 (units processed: 2)
ManagedPoolThread #4 16:13:33 INFO  Job ended: Index_Update_IndexName=sitecore_list_index (units processed: )
ManagedPoolThread #8 16:13:33 INFO  Job started: Index_Update_IndexName=sitecore_list_index

UPDATE 2: Turns out that the order of operations is first indexing, which isn't the issue. Then the next operation is something to do with the EventQueue table. List Manager puts entries in it for processing of something. So if EQ table is jammed, this runs long. But I still don't understand that. So question still applies.

Answer 1

It has been determined that List Manager in Sitecore 8.1 is not as efficient as it could be. Sitecore Support KB Reference Number 431777 assists with List Manager issues. With this installed, List Manager seems to function better, albeit still not as fast.

Instead, the answer from Sitecore Support is that List Manager as a product is being overhauled and rewritten.

Speculation is that there may be a List Manager Hotfix/Update out of cycle before Sitecore 8.3 is released, however, normally List Manager is released with updates on the same schedule as the over all product.

How does this tie into EXM?

EXM uses all List Manager functions now to manage recipient lists. So if you're struggling with lists like I was, contact Sitecore Support and explain that you need all of the List Manager KB's as well as the EXM 3.3 Hotfix.

Answer 2

I'm taking a stab in the dark, but it sounds like this is an issue with the EventQueue getting too large. If as you say an event is generated in the EventQueue for every user being imported, then we're looking at 90,000 rows being generated for every import.

These rows will only be removed when Sitecore's EventQueue CleanUp agent runs, and by default, this is set to 1 day:

<agent type="Sitecore.Tasks.CleanupEventQueue, Sitecore.Kernel" method="Run" interval="04:00:00">
  <DaysToKeep>1</DaysToKeep>
</agent>

Naturally, if this is running multiple times per day, this table can get quite large, and further EventQueue processing will take longer due to the number of entries being sorted and processed in DESC order.

As @g-killian recommended, either try disabling the EventQueue entirely to see if that helps, or decrease the amount of time that entries are kept alive by changing the CleanupEventQueue interval. You should be able to use a value less than 1 day by specifying an <intervalToKeep>hh:mm:ss</intervalToKeep> value instead of <DaysToKeep />.

Answer 3

What I normally did was, index the sitecore_list_index manually (from sitecore desktop/launchpad > control panel > Index Manager) and then manually unlock the list in list manager.

Answer 4

We've just gone through this whole process.

In our case we had an implementation of Lucene indexing rather than the Solr method (multi server set up). We also found we had a rather large bottleneck in our SQL and MongoDB. They just couldn't process the data that was getting thrown at them.

Before implementing Solr, and upgrading our MongoDB and SQL servers (they were Azure hosted VM's):-

1. Emails dispatched at 1-2 emails per second
2. Indexing of lists would take >1hr for list of 10,000 contacts.

Since implementing these changes:-

1. Emails dispatch at 30 emails p/s
2. Lists of 100,000 index in around 4 minutes.

Our azure server upgrades were purely additional iOPS and faster discs rather than more processing power or RAM.

There is also an hotfix from Sitecore that stops list locking - but emails will still only send to the people so far indexed.