17

In Sitecore 8.X, the List Manager is the primary tool used to import contacts and create audience lists for Sitecore. This is a dependency in EXM as well for creating message dispatches against a List Manager list.

When a list is created, it goes through a couple steps:

  1. Sitecore creates the stub of the list as an item in Sitecore.
  2. Sitecore then locks the list from any pending changes.
  3. During the lock phase:
    • Sitecore Creates the xDB contact.
    • Sitecore adds to the sitecore_list_index the list information.
  4. Once the lock phase is complete, Sitecore unlocks the list and it can be used for consumption. enter image description here

The main issue at hand is that for some reason, writing to the sitecore_list_index takes an enormous amount of time. Creating a list of 2 users has thus taken 5 minutes. When I have a CSV of 90,000 that I need to upload.

A little bit about our indexing. We are using Solr Server 6 for all indexes. (Yes, we know Solr 6 isn't an approved version, but so far, we not have any issues and all tests are coming back very positive. The master index builds very fast.)

The sitecore_list_index is very small. enter image description here

Question: How do I improve this experience? Am I even able to?

UPDATE 1: I don't think it's Indexes. According to the log, the index is getting processed and updated almost immediately.

ManagedPoolThread #11 16:13:33 INFO  Job started: Index_Update_IndexName=sitecore_list_index
ManagedPoolThread #9 16:13:33 INFO  Job ended: Index_Update_IndexName=sitecore_master_index (units processed: )
ManagedPoolThread #9 16:13:33 INFO  Job started: Index_Update_IndexName=sitecore_master_index
ManagedPoolThread #17 16:13:33 INFO  Job started: CONTACT_BULK_UPDATE_ListManagementImportContacts_cbaf9d6805a74a509e0413ac2fd751b7
ManagedPoolThread #11 16:13:33 INFO  Job ended: Index_Update_IndexName=sitecore_list_index (units processed: )
ManagedPoolThread #4 16:13:33 INFO  Job started: Index_Update_IndexName=sitecore_list_index
ManagedPoolThread #17 16:13:33 INFO  Job ended: CONTACT_BULK_UPDATE_ListManagementImportContacts_cbaf9d6805a74a509e0413ac2fd751b7 (units processed: 2)
ManagedPoolThread #4 16:13:33 INFO  Job ended: Index_Update_IndexName=sitecore_list_index (units processed: )
ManagedPoolThread #8 16:13:33 INFO  Job started: Index_Update_IndexName=sitecore_list_index

UPDATE 2: Turns out that the order of operations is first indexing, which isn't the issue. Then the next operation is something to do with the EventQueue table. List Manager puts entries in it for processing of something. So if EQ table is jammed, this runs long. But I still don't understand that. So question still applies.

|improve this question
  • Does this also happen when you are indexing other indexes like lucene or Coveo(if you have it)? I'm assuming your pc specs are at minimum with Sitecore's recommendations? SSD? Have you noticed any errors on logs? – Diego Sep 30 '16 at 20:07
  • No. And I'm not sure it's indexing actually. The log show the actual index job only took a mere 5 seconds. But.. the list is taking way longer to unlock. – Pete Navarra Sep 30 '16 at 20:11
  • Added Update #2. – Pete Navarra Sep 30 '16 at 22:57
  • 1
    Can you selectively disable features to try and better isolate this? Disable indexing then test; disable EventQueue then test; try it with a local MongoDB in case that connectivity is at issue? Just because a log message records that indexing is complete, doesn't mean the repercussions (optimizing etc) are done. Let's see . . . that's using ItemBuckets under the covers, so maybe set BucketConfiguration.EnableBucketDebug setting to false in the App_Config/Include/Sitecore.Buckets.config section in the hopes you can short-circuit whatever time consuming process is taking place? – G Killian Oct 7 '16 at 13:04
  • "Creating a list of 2 users has thus taken 5 minutes" is insane – G Killian Oct 7 '16 at 13:05
9

It has been determined that List Manager in Sitecore 8.1 is not as efficient as it could be. Sitecore Support KB Reference Number 431777 assists with List Manager issues. With this installed, List Manager seems to function better, albeit still not as fast.

Instead, the answer from Sitecore Support is that List Manager as a product is being overhauled and rewritten.

Speculation is that there may be a List Manager Hotfix/Update out of cycle before Sitecore 8.3 is released, however, normally List Manager is released with updates on the same schedule as the over all product.

How does this tie into EXM?

EXM uses all List Manager functions now to manage recipient lists. So if you're struggling with lists like I was, contact Sitecore Support and explain that you need all of the List Manager KB's as well as the EXM 3.3 Hotfix.

|improve this answer
  • To follow up on this, there has indeed been a hotfix released - kb.sitecore.net/articles/140560 It unfortunately only works with SOLR, but it sounds like it will be effective if that matches your configuration – Richard Szalay Jan 19 '17 at 5:47
4

I'm taking a stab in the dark, but it sounds like this is an issue with the EventQueue getting too large. If as you say an event is generated in the EventQueue for every user being imported, then we're looking at 90,000 rows being generated for every import.

These rows will only be removed when Sitecore's EventQueue CleanUp agent runs, and by default, this is set to 1 day:

<agent type="Sitecore.Tasks.CleanupEventQueue, Sitecore.Kernel" method="Run" interval="04:00:00">
  <DaysToKeep>1</DaysToKeep>
</agent>

Naturally, if this is running multiple times per day, this table can get quite large, and further EventQueue processing will take longer due to the number of entries being sorted and processed in DESC order.

As @g-killian recommended, either try disabling the EventQueue entirely to see if that helps, or decrease the amount of time that entries are kept alive by changing the CleanupEventQueue interval. You should be able to use a value less than 1 day by specifying an <intervalToKeep>hh:mm:ss</intervalToKeep> value instead of <DaysToKeep />.

|improve this answer
0

What I normally did was, index the sitecore_list_index manually (from sitecore desktop/launchpad > control panel > Index Manager) and then manually unlock the list in list manager.

|improve this answer
  • I can see that as a stop gap. But the client will be using the list upload feature for every mailing. I can't see them wanting to reindex and unlock manually every time. – Pete Navarra Sep 30 '16 at 22:51
0

We've just gone through this whole process.

In our case we had an implementation of Lucene indexing rather than the Solr method (multi server set up). We also found we had a rather large bottleneck in our SQL and MongoDB. They just couldn't process the data that was getting thrown at them.

Before implementing Solr, and upgrading our MongoDB and SQL servers (they were Azure hosted VM's):-

  1. Emails dispatched at 1-2 emails per second
  2. Indexing of lists would take >1hr for list of 10,000 contacts.

Since implementing these changes:-

  1. Emails dispatch at 30 emails p/s
  2. Lists of 100,000 index in around 4 minutes.

Our azure server upgrades were purely additional iOPS and faster discs rather than more processing power or RAM.

There is also an hotfix from Sitecore that stops list locking - but emails will still only send to the people so far indexed.

|improve this answer

Your Answer

By clicking "Post Your Answer", you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.