Sitecore 9 update 1 xDB Data Migration Tool Pipeline Batch Extremely slow/hangs

Question

We have Sitecore 9 Update 1 installed successfully. Both of the below services work fine.

• Sitecore xConnect Search Indexer
• Sitecore Marketing Automation Engine

All required databases are installed properly and respective users are configured properly including collectionuser for Shard tables. collectionuser has membership to db_datareader and db_datawriter and has Connect and Execute permissions on the three Shard databases. We have also reindexed, relinked databases and deployed marketing definitions.

We are trying to use xDB Data Migration tool to transfer all analytics information from our existing Mongo database to Sitecore 9 Sql xDB.

When we trigger the pipeline batch to read MongoDb contacts into xConnect, everything works and we can see that the batches are processed properly. At one point, when it has processed all batches, we are expecting the indexer service to process each batch and save data into Sitecore 9 xDB Sql database tables.

The issue is that, this processing is either extremely slow or it just hangs and we are not able to get any pointers from logs to understand what the issue might be. As an example, we run the xDB data migration pipeline batch and after 12 hours, we see just 300 odd records in the Fact_PageViews table when the total expected count is close to 40K. I don't find any helpful pointers on Sitecore documentation also.

Could one of you shed light on what we might be missing? We are running against a tight deadline to migrate from Sitecore 8.2 to Sitecore 9.

Answer 1

I do not know if my answer will be full however it may help. If you are going to migrate a huge number of contacts and its interactions via xDB Data Migration Tool you need to know how to configure migration properly before run it.

There are several things that can speed up migration.

1. Max Thread Count (default: 1)
2. Minimum Batch Size (default: 100)
3. Page Size (default: 200)
4. Max Request Length (default: 1048576)
5. Resolve Object Settings
6. ...

---

Here is some points.

1. Max Thread Count

Enabling Multiple Threads

By increasing the number of threads used during the migration process you can increase performance, especially when a large number of contacts need to be migrated.

Navigate to Pipelines > MongoDB Contacts to xConnect Migration Pipelines > Read Contacts from MongoDB Pipeline > Iterate MongoDB Contacts and Run Pipelines

Important.

Do not change the Max Thread Count value on any other pipeline step in the contact migration process. Doing so may prevent interaction data from being migrated properly

---

2. Minimum Batch Size

Changing Batch Size

In order to interact with the xConnect collection service as efficiently as possible, the contact migration process only submits data to the xConnect collection service after a certain amount of data is collection. This threshold can be configured on the contact migration process.

Note.

The unit this threshold is measure in an operation. An operation represents a single instruction for the xConnect collection service. Controlling the batch size means specifing the minimum number of operations that must be available before those operations are submitted to the xConnect collection service.

Navigate to Pipelines > MongoDB Contacts to xConnect Migration Pipelines > Read Contacts from MongoDB Pipeline > Add xConnect Client to Context

---

3. Page Size

Changing Page Size

Number of objects read from mongo per one page.

---

4. Max Request Length

Set Maximum Request Size on xConnect Server

By default, IIS limits the amount of data that can be accepted by the web server. This default setting reduces the effects of certain types of denial-of-service attacks.

This setting is configured in the HTTP runtime settings section in web.config. The property is maxRequestLength:

It is recommended that this value be increased to the maximum value 2147483647 while the migration process is running.

Note

Do not forget to reset this value after the migration process is finished running.

---

5. Resolve Object Settings

Changing Resolve Object settings

That is very impotent to know, before changing these values you should understand and analyze the data you want to migrate. For example, do you need to look for existing object or not. Turning the setting on means send an additional request. Set these values carefully.

---

6. ...

I believe there are others things that involve on performance.

---

Read more here

Installation Guide

Answer 2

One thing that people tend not to read is about the indexes in the MongoDB collection. After creating those indexes the xDB migration is nearly as fast as lightning.

Copy/paste from "xDB Data Migration Tool Installation Guide 2.0.1.pdf"

1.2.3 Add Indexes to MongoDB Collections

xDB Data Migration Tool reads virtually all of the data in the MongoDB data for analytics from Sitecore 8.x. In order to do this efficiently, a couple of indexes must be added. Note Adding these indexes is optional. However, if you choose not to add these indexes, you should expect the data migration process to take longer to complete.

1. Open a mongo shell.
2. Select the database with the xDB data you plan to migrate to Sitecore 9.
3. Enter the following commands: db.getCollection('Devices').createIndex({LastKnownContactId:1}) db.getCollection('Interactions').createIndex({ContactId:1})

Answer 3

Eventually with help from Sitecore support, I figured out what was causing performance degradation. For anyone having the same issue, try making the below config change.

[Your sitecore instance]\App_Config\Sitecore\DeviceDetection.Client\Sitecore.CES.DeviceDetection.config update DeviceDetection.Enabled to false

Why I had to do this is still something I am trying to figure out but at least it got me successfully through the data migration part.

Also, the solution mentioned below did not work for me.

https://kb.sitecore.net/articles/828414

Answer 4

Going to add this for others who find themselves here, as there can be numerous reasons for a slow process. Mine was particularly Mongo related.

First, in addition to Alex Von Wolferen's Answer, the following index is helpful.

db.Outcomes.createIndex({"InteractionId": 1})

Second, be sure that your MongoDB server has the following set in accord with the MongoDB docs. The -l flag was particularly important for these types of full database scans.

-f (file size): unlimited -t (cpu time): unlimited -v (virtual memory): unlimited 2 -l (locked-in-memory size): unlimited -n (open files): 64000 -m (memory size): unlimited 1 -u (processes/threads): 64000

This literally took me from 0 entities submitted in 50 minutes to 5000 a minute.