What happens when MongoDB is down?

Question

We are building an infrastructure for xDB internally and our internal network team asked us what would happen if the MongoDB instances were down.

We, of course are deploying MongoDB with Replica set but they want to be sure it will not bring down the site.

I remember from somewhere that Sitecore will cache the entries locally on the CD instances until the MongoDB comes back online and then once up, it will sync. Is this true?

Answer 1

It's also important to know that while Mongo is not available, analytic data is being collected. Sitecore will store the data to the file system until such time the collection server comes back online.

From Akinori Taira @ Sitecore:

In the event that the collections database is unavailable, there is a special ‘Submit Queue’ mechanism that flushes captured data to the local hard drive (the ‘Data\Submit Queue’ folder by default). When the collections database comes back online, a background worker process submits the data from the ‘Submit Queue’ on disk.

Sitecore 7.5 and 8 have a key in the Sitecore.Analytics config that can control Sitecore's behavior if Mongo is unreachable for any reason (best I can tell, this has been removed as of 8.1):

<setting name="Analytics.FailOnDatabaseErrors" value="false" />

The submit queue is enabled/disabled from the Sitecore.Analytics.Tracking config file's UseSubmitQueue key:

 <!--  ANALYTICS USE SUBMIT QUEUE
        Specifies if the submit queue should be used when the contact or session submit operations cannot access the database.
        If set to true, the contact or session is queued until the database is backed up (see the <submitQueue> section).
        Default: true
  -->
  <setting name="Analytics.UseSubmitQueue" value="true" />

The background task that will flush this to Mongo is configured in the Sitecore.Analytics.Tracking.Database.config's SubmitQueue node:

<submitQueue>
  <backgroundService type="Sitecore.Analytics.SubmitQueueService, Sitecore.Analytics">
    <!-- Service wakeup interval in seconds. -->
    <Interval>60</Interval>
  </backgroundService>
</submitQueue>

Other reading/Sources:

Sitecore Climber answered this quite well here: https://stackoverflow.com/questions/39153295/in-sitecore-how-to-view-the-content-of-the-submit-queue-file

And LonghornTaco's great blog post on this subject: https://citizensitecore.com/2016/07/01/xdb-session-info-and-mongodb-availability/

Answer 2

If the collections database is unavailable, analytics data for a user's session will still be collected and stored in Session. The problem comes when the session times out and Sitecore attempts to flush that data from Session to the collections database.

From Akinori Taira, part of the xDB product team:

In the event that the collections database is unavailable, there is a special ‘Submit Queue’ mechanism that flushes captured data to the local hard drive (the ‘Data\Submit Queue’ folder by default). When the collections database comes back online, a background worker process submits the data from the ‘Submit Queue’ on disk.

As with most everything else in Sitecore, this mechanism is controlled by entries, scattered across a few config files.

In Sitecore 8.0 (and its updates) there is a setting called Analytics.FailOnDatabaseErrors in Sitecore.Analytics.config:

    <!-- ANALYTICS FAIL ON DATABASE ERRORS
Note: This is a diagnostics setting that should not be enabled on a live system.
If set to true, web pages crash and show exception details in case
an operation cannot access the database.
If set to false, exceptions are suppressed and logged, and visit data
is queued until the database is back up. See <submitQueue> section.
Default: false
-->
<setting name="Analytics.FailOnDatabaseErrors" value="false" />

However, in Sitecore 8.1+, the property Analytics.UseSubmitQueue is used from Sitecore.Analytics.Tracking.config:

<!--  ANALYTICS USE SUBMIT QUEUE
    Specifies if the submit queue should be used when the contact or session submit operations cannot access the database.
    If set to true, the contact or session is queued until the database is backed up (see the <submitQueue> section).
    Default: true
-->
<setting name="Analytics.UseSubmitQueue" value="true" />

Finally, from the Sitecore.Analytics.Tracking.config and Sitecore.Analytics.Tracking.Database.config, you get the following:

<submitQueue>
    <queue type="Sitecore.Analytics.Data.DataAccess.SubmitQueue.FileSubmitQueue, Sitecore.Analytics" singleInstance="true" />
    <backgroundService type="Sitecore.Analytics.SubmitQueueService, Sitecore.Analytics">
        <!-- Service wakeup interval in seconds. -->
        <Interval>60</Interval>
    </backgroundService>
</submitQueue>

I have not tested this failover scenario myself yet, so your mileage may vary.

Answer 3

If you are about to build a Replica Set based on MongoDB recommendations which says

Three member replica sets provide enough redundancy to survive most network partitions and other system failures

The chances to get MongoDB down will be reduced.

However, just in case an incident happen, the standard configuration for a Replica Set is to provide a level of fault tolerance against the loss of a single database server. And in addition to that, the Secondary will be able to reply the requests until you get the primary back to business.

Answer 4

Below is the 1 main points I know if MongoDb is down, the following occurs

1. Package installation will take too much time to install.

However, it will not bring the site down unless you are querying the MongoDB directly. You may receive error page on the specific page where the query to MongoDB is being executed. Moreover, in order to prevent this, you can make MongoDb run as a service or run it through Sitecore pipeline. Below is the link how you can run MongoDB in Sitecore pipeline

https://blog.istern.dk/2014/10/01/launch-mongo-db-with-sitecore-pipelines/