4

We have Sitecore 9 Update 1 installed successfully. Both of the below services work fine.

  • Sitecore xConnect Search Indexer
  • Sitecore Marketing Automation Engine

All required databases are installed properly and respective users are configured properly including collectionuser for Shard tables. collectionuser has membership to db_datareader and db_datawriter and has Connect and Execute permissions on the three Shard databases. We have also reindexed, relinked databases and deployed marketing definitions.

We are trying to use xDB Data Migration tool to transfer all analytics information from our existing Mongo database to Sitecore 9 Sql xDB.

When we trigger the pipeline batch to read MongoDb contacts into xConnect, everything works and we can see that the batches are processed properly. At one point, when it has processed all batches, we are expecting the indexer service to process each batch and save data into Sitecore 9 xDB Sql database tables.

The issue is that, this processing is either extremely slow or it just hangs and we are not able to get any pointers from logs to understand what the issue might be. As an example, we run the xDB data migration pipeline batch and after 12 hours, we see just 300 odd records in the Fact_PageViews table when the total expected count is close to 40K. I don't find any helpful pointers on Sitecore documentation also.

Could one of you shed light on what we might be missing? We are running against a tight deadline to migrate from Sitecore 8.2 to Sitecore 9.

  • Hi. Can you point xDB Data Migration Tool version? – Vlad Shpak Feb 23 '18 at 20:52
  • We have currently installed the following Data Exchange Framework 2.0.1 rev. 180108, MongoDB Provider for Data Exchange Framework 2.0.1 rev. 180108, Sitecore Provider for Data Exchange Framework 2.0.1 rev. 180108, xConnect Provider for Data Exchange Framework 2.0.1 rev. 180108 and xDB Data Migration Tool for Data Exchange Framework 2.0.1 rev. 180108 – Navneet Pisharodi Feb 23 '18 at 21:02
  • What was your "max thread count" to sync contacts? – Vlad Shpak Feb 23 '18 at 21:16
  • Unfortunately I am not able to access my sitecore environment now since the pipeline batch is in limbo but we did not make any change and are using the default value that was set when we installed the xDB data migration packages. – Navneet Pisharodi Feb 23 '18 at 21:25
  • 1
    I think aggregation is not a part of xDB Data Migration Tool. It means to migrate contacts properly, the tool migrates mongo data through the xConnect. For example if 100 contacts where migrated but agregated only 50 that is issue of xConnect. But I could be wrong here. Sorry. – Vlad Shpak Feb 27 '18 at 15:49
4

I do not know if my answer will be full however it may help. If you are going to migrate a huge number of contacts and its interactions via xDB Data Migration Tool you need to know how to configure migration properly before run it.

There are several things that can speed up migration.

  1. Max Thread Count (default: 1)
  2. Minimum Batch Size (default: 100)
  3. Page Size (default: 200)
  4. Max Request Length (default: 1048576)
  5. Resolve Object Settings
  6. ...

Here is some points.

1. Max Thread Count

Enabling Multiple Threads

By increasing the number of threads used during the migration process you can increase performance, especially when a large number of contacts need to be migrated.

Navigate to Pipelines > MongoDB Contacts to xConnect Migration Pipelines > Read Contacts from MongoDB Pipeline > Iterate MongoDB Contacts and Run Pipelines

Important.

Do not change the Max Thread Count value on any other pipeline step in the contact migration process. Doing so may prevent interaction data from being migrated properly


2. Minimum Batch Size

Changing Batch Size

In order to interact with the xConnect collection service as efficiently as possible, the contact migration process only submits data to the xConnect collection service after a certain amount of data is collection. This threshold can be configured on the contact migration process.

Note.

The unit this threshold is measure in an operation. An operation represents a single instruction for the xConnect collection service. Controlling the batch size means specifing the minimum number of operations that must be available before those operations are submitted to the xConnect collection service.

Navigate to Pipelines > MongoDB Contacts to xConnect Migration Pipelines > Read Contacts from MongoDB Pipeline > Add xConnect Client to Context


3. Page Size

Changing Page Size

Number of objects read from mongo per one page.


4. Max Request Length

Set Maximum Request Size on xConnect Server

By default, IIS limits the amount of data that can be accepted by the web server. This default setting reduces the effects of certain types of denial-of-service attacks.

This setting is configured in the HTTP runtime settings section in web.config. The property is maxRequestLength:

enter image description here

It is recommended that this value be increased to the maximum value 2147483647 while the migration process is running.

Note

Do not forget to reset this value after the migration process is finished running.


5. Resolve Object Settings

Changing Resolve Object settings

That is very impotent to know, before changing these values you should understand and analyze the data you want to migrate. For example, do you need to look for existing object or not. Turning the setting on means send an additional request. Set these values carefully.

enter image description here


6. ...

I believe there are others things that involve on performance.


Read more here

Installation Guide

  • A late reply @Vlad but I was trying out a few things before coming back to you. From the look of it xDB data migration seems to be quick and fine especially after making the configuration changes you suggested above. I now am at a point where I feel that it is the xconnect processing that is extremely slow. Once xDB Data Migration tool has pulled all the batches from Mongo, the xconnect processing engine should go in, aggregate them into SQL. But processing is painfully slow (e.g in 15 hours, I see 466 rows in [dbo].[Fact_PageViews] table where infact I expect 42K rows. Any thoughts? – Navneet Pisharodi Mar 20 '18 at 12:31
  • @NavneetPisharodi Hi, You can play with configuration settings in Sitecore.Analytics.Processing.Aggregation.Services.config At least you can increase default value for "minimumBatchSize" of "contactWorkDispatcher" – Vlad Shpak Mar 20 '18 at 13:43
2

One thing that people tend not to read is about the indexes in the MongoDB collection. After creating those indexes the xDB migration is nearly as fast as lightning.

Copy/paste from "xDB Data Migration Tool Installation Guide 2.0.1.pdf"

1.2.3 Add Indexes to MongoDB Collections

xDB Data Migration Tool reads virtually all of the data in the MongoDB data for analytics from Sitecore 8.x. In order to do this efficiently, a couple of indexes must be added. Note Adding these indexes is optional. However, if you choose not to add these indexes, you should expect the data migration process to take longer to complete.

  1. Open a mongo shell.
  2. Select the database with the xDB data you plan to migrate to Sitecore 9.
  3. Enter the following commands: db.getCollection('Devices').createIndex({LastKnownContactId:1}) db.getCollection('Interactions').createIndex({ContactId:1})
1

Eventually with help from Sitecore support, I figured out what was causing performance degradation. For anyone having the same issue, try making the below config change.

[Your sitecore instance]\App_Config\Sitecore\DeviceDetection.Client\Sitecore.CES.DeviceDetection.config update DeviceDetection.Enabled to false

Why I had to do this is still something I am trying to figure out but at least it got me successfully through the data migration part.

Also, the solution mentioned below did not work for me.

https://kb.sitecore.net/articles/828414

0

Going to add this for others who find themselves here, as there can be numerous reasons for a slow process. Mine was particularly Mongo related.

First, in addition to Alex Von Wolferen's Answer, the following index is helpful.

db.Outcomes.createIndex({"InteractionId": 1})

Second, be sure that your MongoDB server has the following set in accord with the MongoDB docs. The -l flag was particularly important for these types of full database scans.

-f (file size): unlimited -t (cpu time): unlimited -v (virtual memory): unlimited 2 -l (locked-in-memory size): unlimited -n (open files): 64000 -m (memory size): unlimited 1 -u (processes/threads): 64000

This literally took me from 0 entities submitted in 50 minutes to 5000 a minute.

  • thanks @AlexMayle for enhancing the information. – Navneet Pisharodi Apr 8 at 22:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.