2

Does Sitecore have any documentation on xDB shard sizing?

Per Google analytics we have about 8M visitors per month, 20M sessions per month. All Anonymous users.

Default is 2 shards, max is 18.

Sitecore 9.3 Initial Release. On-Prem. SQL Server 2019 Enterprise DB.

How many Shards should we create?

2
  • Just be aware please that the number of shard can't be changed in the future, once you selected a number you will live with it for good. (please double check this with your Sitecore version) Jan 11, 2021 at 3:21
  • There is a custom tool that allows you to remap the shards to any number though it is code contributed by the community and not endorsed by Sitecore.
    – jrap
    Jan 13, 2021 at 11:56

3 Answers 3

2

It all depends on how you like to setup environment. My advice would be to go for at least eight shards. You might ask why? Because with change tracking enabled (that is what Sitecore uses to be able to collect xdb data while rebuilding your xDB index https://doc.sitecore.com/developers/93/sitecore-experience-platform/en/tracking-changes-to-the-xdb-index.html) your databases might get rather large when you have that amount of traffic. Also when shards become 'crowded' they tend to generate high CPU and memory usage. This can be dealt with by spreading the shardmaps (ContactIdShardMap, DeviceProfileIdShardMap and ContactIdentifiersIndexShardMap) in more shards. Lower the stress on your systems.

Like Mohamed mentioned, once you have chosen a certain number of shards you have to stick with that number, at least in an on-prem situation. Splitting shards is only possible when running on Azure SQL https://doc.sitecore.com/developers/93/platform-administration-and-architecture/en/split-or-merge-xdb-collection-database-shards.html

2

I'm unsure of the number of shards you require but I can provide some hopefully useful information from my own experience given that the number of required shards is closely related to the volume of data and thus size of the SQL dbs...

I am running Sitecore 10.1 with xDB split between 2 shards (the default). I struggled to get any idea of how big the xDB SQL database shards were going to be so I've just had to go with it and see how I got on.

I started with a clean xDB, rather than migrating from Mongo. It's been in production for just under 2 months now, and after approx 1.2 million sessions, each of the 2 shards is over 10GB, so 20GB in total. The size of the SQL dbs has been increasing linearly (more or less). Here's a screenshot of one of the shard DBs taken from the Azure SQL metrics blade:

enter image description here

This seems very large compared to the previous MongoDB iteration I had running on Sitecore 8.2. That was approx 45GB and had been running for several years, with many, many millions of sessions.

I am going to continue to monitor the situation and sincerely hope that db size growth is not linear, and starts to plateau. There is a slight curve on the graph, so I'm hopeful, but it's too early to say for sure where it will be in ~12 months. If it starts to go into TB of data then it will probably be uneconomical to continue.

Performance does not appear to be an issue with this volume of data split over 2 shards, although I am not utilizing Sitecore Analytics or any anything that particularly consumes the xDB shards for data aggregation. My use case is more about individual contact profiles and personalization. CPU, Data IO and Log IO are all basically 0% on a general purpose 4vCore elastic pool.

0

I would recommend to go with more shards as by time the data volumes within the shards will grow up so fast. Having multiple shards will improve scalability when storing and accessing large volumes of data.

For many applications, creating a larger number of small shards can be more efficient than having a small number of large shards because they can offer increased opportunities for load balancing.

Check this documentation, it will help you to decide:

https://docs.microsoft.com/en-us/azure/architecture/patterns/sharding

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.