5

Overview

During a disaster recovery test of our environment we make mongoDB unavailable to Sitecore with the expectation that Sitecore will continue to function normally due to the Configuration noted below per Sitecore documentation.

Instead what happens is the website throws exceptions making itself unavailable. Additionaly, I can see a file created on the local CD server in the Data/Submit Queue folder, however after restoring mongoDB these files remain there rather than disappear as expected from being processed by Sitecore.

Configuration

We are currently running Sitecore.NET 8.2 (rev. 170728) with a CM server and several CD servers. I validated the following configuration on all of our CD servers:

Submit Queue Enabled:

<submitQueue patch:source="Sitecore.Analytics.Tracking.config">
  <queue type="Sitecore.Analytics.Data.DataAccess.SubmitQueue.FileSubmitQueue, Sitecore.Analytics" singleInstance="true">
    <param desc="folderPath">$(dataFolder)/Submit Queue</param>
  </queue>

  <backgroundService type="Sitecore.Analytics.SubmitQueueService, Sitecore.Analytics">
    <!--  Service wakeup interval in seconds.  -->
    <Interval>60</Interval>
  </backgroundService>
</submitQueue>

mongoDB set not to fail on read errors:

<mongo recoveryTimeout="60">
  <driver type="Sitecore.Analytics.Data.DataAccess.MongoDb.MongoDbDriver, Sitecore.Analytics.MongoDB">
    <param desc="connectionString">$(0)</param>
    <param desc="failOnReadErrors">false</param>
  </driver>
</mongo>

Exceptions

During the DR testing the CM servers continues to function online as expected, whereas the all CDs being to throw exceptions. The following I believe to be the relevant problem from the Sitecore log:

ERROR Cannot create tracker.
Exception: System.TimeoutException
Message: A timeout occured after 30000ms selecting a server using CompositeServerSelector{ Selectors = WritableServerSelector, LatencyLimitingServerSelector{ AllowedLatencyRange = 00:00:00.0150000 } }. Client view of cluster state is { ClusterId : "1", ConnectionMode : "ReplicaSet", Type : "ReplicaSet", State : "Connected", Servers : [{ ServerId: "{ ClusterId : 1, EndPoint : "Unspecified/mongoserver1:27017" }", EndPoint: "Unspecified/mongoserver1:27017", State: "Disconnected", Type: "Unknown", HeartbeatException: "MongoDB.Driver.MongoConnectionException: An exception occurred while opening a connection to the server. ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 123.123.123.123:27017
   at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at MongoDB.Driver.Core.Connections.TcpStreamFactory.d__7.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at MongoDB.Driver.Core.Connections.TcpStreamFactory.d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at MongoDB.Driver.Core.Connections.BinaryConnection.d__48.MoveNext()
   --- End of inner exception stack trace ---
   at MongoDB.Driver.Core.Connections.BinaryConnection.d__48.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at MongoDB.Driver.Core.Servers.ServerMonitor.d__27.MoveNext()" }, { ServerId: "{ ClusterId : 1, EndPoint : "Unspecified/mongoserver2:27017" }", EndPoint: "Unspecified/mongoserver2:27017", State: "Disconnected", Type: "Unknown", HeartbeatException: "MongoDB.Driver.MongoConnectionException: An exception occurred while opening a connection to the server. ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 124.124.124.124:27017
   at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at MongoDB.Driver.Core.Connections.TcpStreamFactory.d__7.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at MongoDB.Driver.Core.Connections.TcpStreamFactory.d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at MongoDB.Driver.Core.Connections.BinaryConnection.d__48.MoveNext()
   --- End of inner exception stack trace ---
   at MongoDB.Driver.Core.Connections.BinaryConnection.d__48.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at MongoDB.Driver.Core.Servers.ServerMonitor.d__27.MoveNext()" }, { ServerId: "{ ClusterId : 1, EndPoint : "Unspecified/mongoserver3:27017" }", EndPoint: "Unspecified/mongoserver3:27017", State: "Connected", Type: "ReplicaSetSecondary", WireVersionRange: "[0, 4]" }] }.
Source: MongoDB.Driver.Core
   at MongoDB.Driver.Core.Clusters.Cluster.ThrowTimeoutException(IServerSelector selector, ClusterDescription description)
   at MongoDB.Driver.Core.Clusters.Cluster.WaitForDescriptionChangedHelper.HandleCompletedTask(Task completedTask)
   at MongoDB.Driver.Core.Clusters.Cluster.WaitForDescriptionChanged(IServerSelector selector, ClusterDescription description, Task descriptionChangedTask, TimeSpan timeout, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Clusters.Cluster.SelectServer(IServerSelector selector, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Bindings.WritableServerBinding.GetReadChannelSource(CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Bindings.ReadWriteBindingHandle.GetReadChannelSource(CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.FindOperation`1.Execute(IReadBinding binding, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollection.ExecuteReadOperation[TResult](IReadOperation`1 operation, ReadPreference readPreference)
   at MongoDB.Driver.MongoCollection.FindOneAs[TDocument](FindOneArgs args)
   at Sitecore.Analytics.Data.DataAccess.MongoDb.MongoDbCollection.c__DisplayClass2a`1.b__29()
   at Sitecore.Analytics.Data.DataAccess.MongoDb.MongoDbCollection.Execute(Action action, ExceptionBehavior exceptionBehavior)
   at Sitecore.Analytics.Data.DataAccess.MongoDb.MongoDbCollection.FindOneAs[TDocument](IMongoQuery query, ExceptionBehavior exceptionBehavior)
   at Sitecore.Analytics.Data.DataAccess.MongoDb.MongoDbContactStorage.LoadContactIdentifier(ID id)
   at Sitecore.Analytics.Data.DataAccess.MongoDb.MongoDbDataAdapterProvider.LoadContactIdentifierValue(ID contactId)
   at Sitecore.Analytics.Data.DataAccess.MongoDb.MongoDbDataAdapterProvider.TryLoadContact(ID id, IContactFactory factory, LeaseOwner leaseOwner, TimeSpan leaseDuration)
   at Sitecore.Analytics.Data.ContactRepository.TryLoadContact(Guid id, LeaseOwner leaseOwner, TimeSpan leaseDuration)
   at Sitecore.Analytics.Tracking.ContactManager.TryLoadContact(Guid contactId, Int32 lockDurationMinutes, Boolean exclusive)
   at Sitecore.Analytics.Pipelines.EnsureSessionContext.LoadContact.Process(InitializeTrackerArgs args)
   at (Object , Object[] )
   at Sitecore.Pipelines.CorePipeline.Run(PipelineArgs args)
   at Sitecore.Pipelines.DefaultCorePipelineManager.Run(String pipelineName, PipelineArgs args, String pipelineDomain)
   at Sitecore.Analytics.Pipelines.EnsureSessionContext.EnsureSessionContextPipeline.Run(InitializeTrackerArgs args)
   at Sitecore.Analytics.DefaultTracker.EnsureSessionContext()
   at Sitecore.Analytics.Pipelines.CreateTracker.GetTracker.Process(CreateTrackerArgs args)
   at (Object , Object[] )
   at Sitecore.Pipelines.CorePipeline.Run(PipelineArgs args)
   at Sitecore.Pipelines.DefaultCorePipelineManager.Run(String pipelineName, PipelineArgs args, String pipelineDomain)
   at Sitecore.Analytics.Tracker.Initialize()
ERROR Application error.
Exception: System.InvalidOperationException
Message: Tracker.Current is not initialized
Source: Sitecore.Analytics
   at Sitecore.Analytics.Pipelines.StartAnalytics.StartTracking.Process(PipelineArgs args)
   at (Object , Object[] )
   at Sitecore.Pipelines.CorePipeline.Run(PipelineArgs args)
   at Sitecore.Pipelines.DefaultCorePipelineManager.Run(String pipelineName, PipelineArgs args, Boolean failIfNotExists)
   at Sitecore.Analytics.Pipelines.StartAnalytics.StartAnalyticsPipeline.Run()
   at Sitecore.Mvc.Analytics.Pipelines.MvcEvents.RequestBegin.StartTracking.Process(RequestBeginArgs args)
   at (Object , Object[] )
   at Sitecore.Pipelines.CorePipeline.Run(PipelineArgs args)
   at Sitecore.Pipelines.DefaultCorePipelineManager.Run(String pipelineName, PipelineArgs args, String pipelineDomain)
   at Sitecore.Mvc.Pipelines.PipelineService.RunPipeline[TArgs](String pipelineName, TArgs args)
   at Sitecore.Mvc.Routing.RouteHttpHandler.BeginProcessRequest(HttpContext context, AsyncCallback cb, Object extraData)
   at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
   at System.Web.HttpApplication.ExecuteStepImpl(IExecutionStep step)
   at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)

I would think this to be a separate issue, though since it occurs during the same timeframe of mongoDB being down I am posting this exception here as well. I am not sure why WFFM would no longer be able to obtain a session (we use SQL for shared sessions).

WARN [WFFM] CurrentSession  is not initialized
ERROR SessionEndPipeline failed.
Exception: System.NullReferenceException
Message: Object reference not set to an instance of an object.
Source: Sitecore.Analytics.Automation
   at Sitecore.Analytics.Pipelines.VisitEnd.RunAutomation.RunAutomationX(Database database)
   at (Object , Object[] )
   at Sitecore.Pipelines.CorePipeline.Run(PipelineArgs args)
   at Sitecore.Pipelines.DefaultCorePipelineManager.Run(String pipelineName, PipelineArgs args, String pipelineDomain)
   at Sitecore.Analytics.Pipelines.VisitEnd.VisitEndPipeline.Run(VisitEndArgs args)
   at (Object , Object[] )
   at Sitecore.Pipelines.CorePipeline.Run(PipelineArgs args)
   at Sitecore.Pipelines.DefaultCorePipelineManager.Run(String pipelineName, PipelineArgs args, String pipelineDomain)
   at Sitecore.Pipelines.EndSession.SessionEndPipeline.Run(SessionEndArgs sessionEndArgs)
   at Sitecore.Web.Application.RaiseSessionEndEvent(HttpApplication context)

The question here being - why does Sitecore fail when mongoDB goes down when the configuration is defined to allow Sitecore to continue to function?

  • 1
    Have you contacted Sitecore Support? – Marek Musielak Apr 26 '18 at 19:09
  • Hi @TShiner, I am also facing the same issue. have you got any further details or fixes on how this issue can be solved? – Sreekrishnan Jul 4 '18 at 2:18
  • @Sreekrishnan We were unable to resolve ourselves. We created a ticket with Sitecore, who supplied a patch to resolve the issue. I would contact Sitecore and reference support patch 147407. – TShiner Jul 6 '18 at 13:07
  • Were you able to solve the issue? It seems the website you provided in the beginning of your question now contains the appropriate patches accordingly to your Sitecore version. – Vinicius Deschamps Aug 6 '18 at 14:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.