Sizing MongoDB Servers for xDB

Question

Is there guidance on sizing the Mongo DB servers when running xDB on premises? I've found some guidance on sizing MongoDB for session state (link) but can't find anything on the analytics side.

To be clear, I'm not looking for "you need XXGB of RAM and XXXGB of disk" but more of what variable go into what sort of formula for sizing.

Answer 1

Nonlinear Digital did a little rule of thumb for 7.5, I guess it still holds true(ish) now - they said:

"As a basic rule of thumb, Sitecore calculates diskspace sizing projections using 5KB per interaction and 2.5KB per identified contact and these two items make up 80% of the diskspace"

So if an interaction is essentially a user session you should be able to roughly calculate usage. For example for a (small) site we're launching we roughly figured the following:

Google Analytics said it had 12,916 visits per month With 9,620 unique users

So:

5kB x 12,916 = 65MB per month (assuming each visit is an "interaction")
+
2.5kB x 9,620 = 25MB per month (assuming each user gets stored as a new "contact")
= 90MB
* 20%
= 109 MB

So around 100MB per month.

This is very finger in the air calculation and I've not validated the figures that NonLinear talk about but it is a good start for you.

References

http://www.nonlinearcreations.com/Digital/how-we-think/whitepapers/Whitepaper-Planning-your-Sitecore-xDB-infrastructure.aspx

Answer 2

Sitecore recommends here the minimum hardware for planning an xDB with Session State as follows:

MongoDB

• 2 Collection database (MongoDB) servers
• 4 x CPU E5 2650 v2 processors
• 16GB RAM (the more the better)
• 100GB SSD disk

In addition to that, I would recommend a separate disk for the Operating System, and leave the 100GB SSD disk just for the collections.

Although, MongoDB has math which will drive you to better estimates here

The amount of RAM you need depends on several factors, including but not limited to:

• The relationship between database storage and working set.
• The operating system’s cache strategy for LRU (Least Recently Used)
• The impact of journaling
• The number or rate of page faults and other MongoDB Cloud Manager gauges to detect when you need more RAM
• Each database connection thread will need up to 1 MB of RAM.

Please feel free to comment if any clarification or additional question needed.

Answer 3

Our best practice is to set up a mongodb replica set consisting of 3 servers (1 primary and 2 secondary). Setting up a cluster of only 2 servers, is not reliable enough as a failover system because it only has majority of 1 (more of that later).

My recommendation:

• 3 database servers (MongoDB) set-up as a replica set
• 4 x CPU E5 2650 v2 processors
• 8GB RAM
• 100GB HDD (dedicated to db storage)

If you have a webshop with 10.000 of visitors every day, you might want to boost up the RAM in your machines, but if this isn't the case, the machines are practically doing nothing.

The HDD depends on which storage mode you are using for your MongoDB. If you use the most compressed storage, you won't need that much storage at all.

To continue the part I was previously talking about, in a replica set, the MongoDB machines act as a "family". If a primary wants to resign and wants become a secondary machine, a voting will take place for the other machines about who will become the primary machines. A replica set will always act as a set containing a primary and a secondary machine. If one of these is not present, the replica set will fall apart causing real trouble for your site.

Answer 4

Sitecore's use of Mongo isn't typical so be careful when looking at RAM recommendations for Mongo. My advice is to size your Mongo data servers as you would SQL servers.

As far as data growth, every site is different so there isn't an exact formula. However, one I've used in the past is this:

(Total pageviews / pages per session) * 4kb = total for number of interactions

Number of unique sessions * 2.5kb = total for contacts

Over a duration, adding the total for interactions to the total for contacts will give you the amount of disk space you need.

Another consideration when doing these calculations is your data archiving strategy - especially around anonymous users, how long do you keep that data?

Answer 5

For disk size, this is quite difficult as it is affected by the amount of visits and interactions you have. Hard to give you exact numbers, although you might easily get to some GB.

Can't give you the magic formula either, but I can give you one advice to keep your needed disk size to a reasonable amount: take a good look at the bots that visit your site, including your own. With "your own", I mean a site crawler if you are using one or a monitoring tool that checks (visits) your site. Blocking those has saved us several GB's (we forgot this only once...). You can patch the Sitecore.Analytics.ExcludeRobots.config to do this.