Sitecore Stack Exchange is a question and answer site for developers and end users of the Sitecore CMS and multichannel marketing software. It only takes a minute to sign up.
Sign up to join this community
Does Sitecore have any documentation on xDB shard sizing?
Per Google analytics we have about 8M visitors per month, 20M sessions per month. All Anonymous users.
Default is 2 shards, max is 18.
Sitecore 9.3 Initial Release. On-Prem. SQL Server 2019 Enterprise DB.
How many Shards should we create?
It all depends on how you like to setup environment. My advice would be to go for at least eight shards. You might ask why? Because with change tracking enabled (that is what Sitecore uses to be able to collect xdb data while rebuilding your xDB index https://doc.sitecore.com/developers/93/sitecore-experience-platform/en/tracking-changes-to-the-xdb-index.html) your databases might get rather large when you have that amount of traffic. Also when shards become 'crowded' they tend to generate high CPU and memory usage. This can be dealt with by spreading the shardmaps (ContactIdShardMap, DeviceProfileIdShardMap and ContactIdentifiersIndexShardMap) in more shards. Lower the stress on your systems.
Like Mohamed mentioned, once you have chosen a certain number of shards you have to stick with that number, at least in an on-prem situation. Splitting shards is only possible when running on Azure SQL https://doc.sitecore.com/developers/93/platform-administration-and-architecture/en/split-or-merge-xdb-collection-database-shards.html
I'm unsure of the number of shards you require but I can provide some hopefully useful information from my own experience given that the number of required shards is closely related to the volume of data and thus size of the SQL dbs...
I am running Sitecore 10.1 with xDB split between 2 shards (the default). I struggled to get any idea of how big the xDB SQL database shards were going to be so I've just had to go with it and see how I got on.
I started with a clean xDB, rather than migrating from Mongo. It's been in production for just under 2 months now, and after approx 1.2 million sessions, each of the 2 shards is over 10GB, so 20GB in total. The size of the SQL dbs has been increasing linearly (more or less). Here's a screenshot of one of the shard DBs taken from the Azure SQL metrics blade:
This seems very large compared to the previous MongoDB iteration I had running on Sitecore 8.2. That was approx 45GB and had been running for several years, with many, many millions of sessions.
I am going to continue to monitor the situation and sincerely hope that db size growth is not linear, and starts to plateau. There is a slight curve on the graph, so I'm hopeful, but it's too early to say for sure where it will be in ~12 months. If it starts to go into TB of data then it will probably be uneconomical to continue.
Performance does not appear to be an issue with this volume of data split over 2 shards, although I am not utilizing Sitecore Analytics or any anything that particularly consumes the xDB shards for data aggregation. My use case is more about individual contact profiles and personalization. CPU, Data IO and Log IO are all basically 0% on a general purpose 4vCore elastic pool.
I would recommend to go with more shards as by time the data volumes within the shards will grow up so fast. Having multiple shards will improve scalability when storing and accessing large volumes of data.
For many applications, creating a larger number of small shards can be more efficient than having a small number of large shards because they can offer increased opportunities for load balancing.
Check this documentation, it will help you to decide:
https://docs.microsoft.com/en-us/azure/architecture/patterns/sharding
