10

Is there guidance on sizing the Mongo DB servers when running xDB on premises? I've found some guidance on sizing MongoDB for session state (link) but can't find anything on the analytics side.

To be clear, I'm not looking for "you need XXGB of RAM and XXXGB of disk" but more of what variable go into what sort of formula for sizing.

  • Are you looking for CPU/Memory, disk, all of the above? – techphoria414 Oct 12 '16 at 15:04
  • Yeah, all of the above, especially disk. The session state guidance gives some information that the typical visit is 30k, so multiply that by concurrent visits to get a rough idea of the db size. I get much of this is going "to depend" but some rules of thumb would be nice. – Derek Dysart Oct 12 '16 at 15:07
  • @ddysart Do you know how many pages your visitors will open per visit on average? Do you know how much facet data you are going to store for every contact? Do you have some amount of monthly visits you're expecting? Are you going to use sharding? Replication? There are indeed too many variables to give any sort of guideline. – Dmytro Shevchenko Oct 12 '16 at 15:15
9

Nonlinear Digital did a little rule of thumb for 7.5, I guess it still holds true(ish) now - they said:

"As a basic rule of thumb, Sitecore calculates diskspace sizing projections using 5KB per interaction and 2.5KB per identified contact and these two items make up 80% of the diskspace"

So if an interaction is essentially a user session you should be able to roughly calculate usage. For example for a (small) site we're launching we roughly figured the following:

Google Analytics said it had 12,916 visits per month With 9,620 unique users

So:

5kB x 12,916 = 65MB per month (assuming each visit is an "interaction")
+
2.5kB x 9,620 = 25MB per month (assuming each user gets stored as a new "contact")
= 90MB
* 20%
= 109 MB

So around 100MB per month.

This is very finger in the air calculation and I've not validated the figures that NonLinear talk about but it is a good start for you.

References

http://www.nonlinearcreations.com/Digital/how-we-think/whitepapers/Whitepaper-Planning-your-Sitecore-xDB-infrastructure.aspx

15

Sitecore recommends here the minimum hardware for planning an xDB with Session State as follows:

MongoDB

  • 2 Collection database (MongoDB) servers
  • 4 x CPU E5 2650 v2 processors
  • 16GB RAM (the more the better)
  • 100GB SSD disk

In addition to that, I would recommend a separate disk for the Operating System, and leave the 100GB SSD disk just for the collections.

Although, MongoDB has math which will drive you to better estimates here

The amount of RAM you need depends on several factors, including but not limited to:

  • The relationship between database storage and working set.
  • The operating system’s cache strategy for LRU (Least Recently Used)
  • The impact of journaling
  • The number or rate of page faults and other MongoDB Cloud Manager gauges to detect when you need more RAM
  • Each database connection thread will need up to 1 MB of RAM.

Please feel free to comment if any clarification or additional question needed.

  • 1
    Answers that are simply links off to other sites are discouraged. Can you summarize the answer here and provide the links as a footnote? – techphoria414 Oct 12 '16 at 15:08
  • @techphoria414 thanks for the feedback, hope the changes reflect what you said. And looking forward to help since first answer :) – Vinicius Deschamps Oct 12 '16 at 15:17
3

Our best practice is to set up a mongodb replica set consisting of 3 servers (1 primary and 2 secondary). Setting up a cluster of only 2 servers, is not reliable enough as a failover system because it only has majority of 1 (more of that later).

My recommendation:

  • 3 database servers (MongoDB) set-up as a replica set
  • 4 x CPU E5 2650 v2 processors
  • 8GB RAM
  • 100GB HDD (dedicated to db storage)

If you have a webshop with 10.000 of visitors every day, you might want to boost up the RAM in your machines, but if this isn't the case, the machines are practically doing nothing.

The HDD depends on which storage mode you are using for your MongoDB. If you use the most compressed storage, you won't need that much storage at all.

To continue the part I was previously talking about, in a replica set, the MongoDB machines act as a "family". If a primary wants to resign and wants become a secondary machine, a voting will take place for the other machines about who will become the primary machines. A replica set will always act as a set containing a primary and a secondary machine. If one of these is not present, the replica set will fall apart causing real trouble for your site.

2

Sitecore's use of Mongo isn't typical so be careful when looking at RAM recommendations for Mongo. My advice is to size your Mongo data servers as you would SQL servers.

As far as data growth, every site is different so there isn't an exact formula. However, one I've used in the past is this:

(Total pageviews / pages per session) * 4kb = total for number of interactions

Number of unique sessions * 2.5kb = total for contacts

Over a duration, adding the total for interactions to the total for contacts will give you the amount of disk space you need.

Another consideration when doing these calculations is your data archiving strategy - especially around anonymous users, how long do you keep that data?

1

For disk size, this is quite difficult as it is affected by the amount of visits and interactions you have. Hard to give you exact numbers, although you might easily get to some GB.

Can't give you the magic formula either, but I can give you one advice to keep your needed disk size to a reasonable amount: take a good look at the bots that visit your site, including your own. With "your own", I mean a site crawler if you are using one or a monitoring tool that checks (visits) your site. Blocking those has saved us several GB's (we forgot this only once...). You can patch the Sitecore.Analytics.ExcludeRobots.config to do this.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.