Skip to main content
added 87 characters in body
Source Link
Vlad Shpak
  • 1.8k
  • 9
  • 12

I do not know if my answer will be full however it may help. If you are going to migrate a huge number of contacts and its interactions via xDB Data Migration Tool you need to know how to configure migration properly before run it.

There are several things that can speed up migration.

  1. Max Thread Count (default: 1)
  2. Minimum Batch Size (default: 100)
  3. Page Size (default: 200)
  4. Max Request Length (default: 1048576)
  5. Resolve Object Settings
  6. ...

Here is some points.

1. Max Thread Count

Enabling Multiple Threads

By increasing the number of threads used during the migration process you can increase performance, especially when a large number of contacts need to be migrated.

Navigate to Pipelines > MongoDB Contacts to xConnect Migration Pipelines > Read Contacts from MongoDB Pipeline > Iterate MongoDB Contacts and Run Pipelines

Important.

Do not change the Max Thread Count value on any other pipeline step in the contact migration process. Doing so may prevent interaction data from being migrated properly


2. Minimum Batch Size

Changing Batch Size

In order to interact with the xConnect collection service as efficiently as possible, the contact migration process only submits data to the xConnect collection service after a certain amount of data is collection. This threshold can be configured on the contact migration process.

Note.

The unit this threshold is measure in an operation. An operation represents a single instruction for the xConnect collection service. Controlling the batch size means specifing the minimum number of operations that must be available before those operations are submitted to the xConnect collection service.

Navigate *to Pipelines > MongoDB Contacts to xConnect Migration Pipelines > Read

Contacts from MongoDB Pipeline > Add xConnect Client to Context*


3. Page Size

Changing Page Size

Number of objects read from mongo per one page.


4. Max Request Length

Set Maximum Request Size on xConnect Server

By default, IIS limits the amount of data that can be accepted by the web server. This default setting reduces the effects of certain types of denial-of-service attacks.

This setting is configured in the HTTP runtime settings section in web.config. The property is maxRequestLength:

enter image description here

It is recommended that this value be increased to the maximum value 2147483647 while the migration process is running.

Note

Do not forget to reset this value after the migration process is finished running.


5. Resolve Object Settings

Changing Resolve Object settings

That is very impotent to know, before changing these values you should understand and analyze the data you want to migrate. For example, do you need to look for existing object or not. Turning the setting on means send an additional request. Set these values carefully.

enter image description here


6. ...

I believe there are others things that involve on performance.


Read more here

Installation Guide

I do not know if my answer will be full however it may help. If you are going to migrate a huge number of contacts and its interactions via xDB Data Migration Tool you need to know how to configure migration properly before run it.

There are several things that can speed up migration.

  1. Max Thread Count (default: 1)
  2. Minimum Batch Size (default: 100)
  3. Page Size (default: 200)
  4. Max Request Length (default: 1048576)
  5. Resolve Object Settings
  6. ...

Here is some points.

1. Max Thread Count

Enabling Multiple Threads

By increasing the number of threads used during the migration process you can increase performance, especially when a large number of contacts need to be migrated.

Navigate to Pipelines > MongoDB Contacts to xConnect Migration Pipelines > Read Contacts from MongoDB Pipeline > Iterate MongoDB Contacts and Run Pipelines

Important.

Do not change the Max Thread Count value on any other pipeline step in the contact migration process. Doing so may prevent interaction data from being migrated properly


2. Minimum Batch Size

Changing Batch Size

In order to interact with the xConnect collection service as efficiently as possible, the contact migration process only submits data to the xConnect collection service after a certain amount of data is collection. This threshold can be configured on the contact migration process.

Note.

The unit this threshold is measure in an operation. An operation represents a single instruction for the xConnect collection service. Controlling the batch size means specifing the minimum number of operations that must be available before those operations are submitted to the xConnect collection service.

Navigate *to Pipelines > MongoDB Contacts to xConnect Migration Pipelines > Read

Contacts from MongoDB Pipeline > Add xConnect Client to Context*


3. Page Size

Changing Page Size

Number of objects read from mongo per one page.


4. Max Request Length

Set Maximum Request Size on xConnect Server

By default, IIS limits the amount of data that can be accepted by the web server. This default setting reduces the effects of certain types of denial-of-service attacks.

This setting is configured in the HTTP runtime settings section in web.config. The property is maxRequestLength:

enter image description here

It is recommended that this value be increased to the maximum value 2147483647 while the migration process is running.

Note

Do not forget to reset this value after the migration process is finished running.


5. Resolve Object Settings

Changing Resolve Object settings

That is very impotent to know, before changing these values you should understand and analyze the data you want to migrate. For example, do you need to look for existing object or not. Turning the setting on means send an additional request. Set these values carefully.


6. ...

I believe there are others things that involve on performance.


Read more here

Installation Guide

I do not know if my answer will be full however it may help. If you are going to migrate a huge number of contacts and its interactions via xDB Data Migration Tool you need to know how to configure migration properly before run it.

There are several things that can speed up migration.

  1. Max Thread Count (default: 1)
  2. Minimum Batch Size (default: 100)
  3. Page Size (default: 200)
  4. Max Request Length (default: 1048576)
  5. Resolve Object Settings
  6. ...

Here is some points.

1. Max Thread Count

Enabling Multiple Threads

By increasing the number of threads used during the migration process you can increase performance, especially when a large number of contacts need to be migrated.

Navigate to Pipelines > MongoDB Contacts to xConnect Migration Pipelines > Read Contacts from MongoDB Pipeline > Iterate MongoDB Contacts and Run Pipelines

Important.

Do not change the Max Thread Count value on any other pipeline step in the contact migration process. Doing so may prevent interaction data from being migrated properly


2. Minimum Batch Size

Changing Batch Size

In order to interact with the xConnect collection service as efficiently as possible, the contact migration process only submits data to the xConnect collection service after a certain amount of data is collection. This threshold can be configured on the contact migration process.

Note.

The unit this threshold is measure in an operation. An operation represents a single instruction for the xConnect collection service. Controlling the batch size means specifing the minimum number of operations that must be available before those operations are submitted to the xConnect collection service.

Navigate *to Pipelines > MongoDB Contacts to xConnect Migration Pipelines > Read

Contacts from MongoDB Pipeline > Add xConnect Client to Context*


3. Page Size

Changing Page Size

Number of objects read from mongo per one page.


4. Max Request Length

Set Maximum Request Size on xConnect Server

By default, IIS limits the amount of data that can be accepted by the web server. This default setting reduces the effects of certain types of denial-of-service attacks.

This setting is configured in the HTTP runtime settings section in web.config. The property is maxRequestLength:

enter image description here

It is recommended that this value be increased to the maximum value 2147483647 while the migration process is running.

Note

Do not forget to reset this value after the migration process is finished running.


5. Resolve Object Settings

Changing Resolve Object settings

That is very impotent to know, before changing these values you should understand and analyze the data you want to migrate. For example, do you need to look for existing object or not. Turning the setting on means send an additional request. Set these values carefully.

enter image description here


6. ...

I believe there are others things that involve on performance.


Read more here

Installation Guide

Source Link
Vlad Shpak
  • 1.8k
  • 9
  • 12

I do not know if my answer will be full however it may help. If you are going to migrate a huge number of contacts and its interactions via xDB Data Migration Tool you need to know how to configure migration properly before run it.

There are several things that can speed up migration.

  1. Max Thread Count (default: 1)
  2. Minimum Batch Size (default: 100)
  3. Page Size (default: 200)
  4. Max Request Length (default: 1048576)
  5. Resolve Object Settings
  6. ...

Here is some points.

1. Max Thread Count

Enabling Multiple Threads

By increasing the number of threads used during the migration process you can increase performance, especially when a large number of contacts need to be migrated.

Navigate to Pipelines > MongoDB Contacts to xConnect Migration Pipelines > Read Contacts from MongoDB Pipeline > Iterate MongoDB Contacts and Run Pipelines

Important.

Do not change the Max Thread Count value on any other pipeline step in the contact migration process. Doing so may prevent interaction data from being migrated properly


2. Minimum Batch Size

Changing Batch Size

In order to interact with the xConnect collection service as efficiently as possible, the contact migration process only submits data to the xConnect collection service after a certain amount of data is collection. This threshold can be configured on the contact migration process.

Note.

The unit this threshold is measure in an operation. An operation represents a single instruction for the xConnect collection service. Controlling the batch size means specifing the minimum number of operations that must be available before those operations are submitted to the xConnect collection service.

Navigate *to Pipelines > MongoDB Contacts to xConnect Migration Pipelines > Read

Contacts from MongoDB Pipeline > Add xConnect Client to Context*


3. Page Size

Changing Page Size

Number of objects read from mongo per one page.


4. Max Request Length

Set Maximum Request Size on xConnect Server

By default, IIS limits the amount of data that can be accepted by the web server. This default setting reduces the effects of certain types of denial-of-service attacks.

This setting is configured in the HTTP runtime settings section in web.config. The property is maxRequestLength:

enter image description here

It is recommended that this value be increased to the maximum value 2147483647 while the migration process is running.

Note

Do not forget to reset this value after the migration process is finished running.


5. Resolve Object Settings

Changing Resolve Object settings

That is very impotent to know, before changing these values you should understand and analyze the data you want to migrate. For example, do you need to look for existing object or not. Turning the setting on means send an additional request. Set these values carefully.


6. ...

I believe there are others things that involve on performance.


Read more here

Installation Guide