Our problem is that periodically one of the CD servers CPU soars to 100% and IIS needs to be restarted to get it to calm down. While investigating the logs we found the following two stacktraces:
ManagedPoolThread #97 15:45:40 ERROR Error committing data row Pages.ef42d298-9973-4c98-9457-1ad4c4761749 : Concurrency violation: the UpdateCommand affected 0 of the expected 1 records. ManagedPoolThread #97 15:45:40 WARN Errors while committing data. Retrying... System.Exception: Error committing datatable: Pages ---> System.Exception: Commit error: Concurrency violation: the UpdateCommand affected 0 of the expected 1 records. at Sitecore.Analytics.Data.DataAccess.DataAdapters.Sql.SqlDataAdapterProvider.Update(DataTable table, UpdateOptions options) at Sitecore.Analytics.Data.DataAccess.DataAdapters.Sql.SqlDataAdapterProvider.Update(DataTable table) at Sitecore.Analytics.Data.DataAccess.DataAdapters.Sql.SqlDataAdapterProvider.Update(DataSet dataSet) --- End of inner exception stack trace --- at Sitecore.Pipelines.CorePipeline.Run(PipelineArgs args) at Sitecore.Pipelines.CorePipeline.Run(String pipelineName, PipelineArgs args, String pipelineDomain, Boolean failIfNotExists) at Sitecore.Pipelines.CorePipeline.Run(String pipelineName, PipelineArgs args, String pipelineDomain) at Sitecore.Pipelines.CorePipeline.Run(String pipelineName, PipelineArgs args) at Sitecore.Analytics.Pipelines.CommitDataSet.CommitDataSetPipeline.Run(CommitDataSetArgs args) at Sitecore.Analytics.Data.TrackerChanges.__AW_Flush()
For this particular issue we found this knowledge base article (Concurrency violation issues in DMS) which indicated that perhaps a visitor had bounced from one of the load-balanced CDs to the other. The load balancer is configured for sticky sessions but the visitor could have transferred for several reasons like one of the cookies expired, one of the CDs was being taken out of the pool temporarily, etc. Since our scenario lined up with "Scenario 1" in the documentation (Load balancer with 2 or more CDs, and InProc Session State mode), our solution could be to configure a SQLServer based session state.
Another error we saw at the same time on the same CD was one that is pretty familiar when the CDs aren't configured properly or the load balancer isn't using sticky sessions:
34320 21:16:21 ERROR Application error. Exception: System.Web.HttpException Message: Validation of viewstate MAC failed. If this application is hosted by a Web Farm or cluster, ensure that <machineKey> configuration specifies the same validationKey and validation algorithm. AutoGenerate cannot be used in a cluster. See http://go.microsoft.com/fwlink/?LinkID=314055 for more information. Source: System.Web at System.Web.UI.ViewStateException.ThrowError(Exception inner, String persistedState, String errorPageMessage, Boolean macValidationError) at System.Web.UI.ViewStateException.ThrowMacValidationError(Exception inner, String persistedState) at System.Web.UI.ObjectStateFormatter.Deserialize(String inputString, Purpose purpose) at System.Web.UI.ObjectStateFormatter.System.Web.UI.IStateFormatter2.Deserialize(String serializedState, Purpose purpose) at System.Web.UI.Util.DeserializeWithAssert(IStateFormatter2 formatter, String serializedState, Purpose purpose) at System.Web.UI.HiddenFieldPageStatePersister.Load() at System.Web.UI.Page.LoadPageStateFromPersistenceMedium() at System.Web.UI.Page.LoadAllState() at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) at System.Web.UI.Page.ProcessRequest(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) at System.Web.UI.Page.ProcessRequest() at System.Web.UI.Page.ProcessRequestWithNoAssert(HttpContext context) at System.Web.UI.Page.ProcessRequest(HttpContext context) at ASP._layouts_flm_masterpages_baselayout_aspx.__AW_ProcessRequest(HttpContext ) at ASP._layouts_flm_masterpages_baselayout_aspx.ProcessRequest(HttpContext context) at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) Nested Exception Exception: System.Web.UI.ViewStateException Message: Invalid viewstate. Client IP: 10.8.5.172 Port: 44240 Referer: http://xxxxxxxxxxxxxxxxx/Blog/Default.aspx Path: /Blog/Default.aspx User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.109 Safari/537.36 ViewState: /wEPDwUKLTkxNzg0OTU0NQ9kFgQCAQ9kFgJmD2QWBgIDDxYCHgRUZXh0BRsgLSBGaW5hbmNpYWwgTGl0ZXJhY3kgTW9udGhkAgUPFgIfAAXFATxtZXRhIG5hbWU9ImtleXdvcmRzIiBjb250ZW50PSJGaW5hbmNpYWwgYmxvZ3MsIGZpbmFuY2lhbCBibG9nLCBmaW5hbmNpYWwgbGl0ZXJhY3kgYmxvZywgZmluYW5jaWFsIHNlcnZpY2VzIGJsb2csIGZpbmFuY2lhbCBzdWNjZXNzIGJsb2csLCBwZXJzb25hbCBidWRnZXRpbmcgYmxvZywgZmluYW5jaWFsIGxpdGVyYWN5IG1vbnRoIGJsb2ciIC8+ZAIHDxYCHwAF7wE8bWV0YSBuYW1lPSJkZXNjcmlwdGlvbiIgY29udGVudD0iRmluYW5jaWFsIExpdGVyYWN5IE1vbnRoIEJsb2cgYWJvdXQgRmluYW5jaWFsIFN1Y2Nlc3MuICBTdGFydGluZyBvbiB0aGUgZmlyc3QgZGF5IG9mIEZpbmFuY2lhbCBMaXRlcmFjeSBNb250aCAoQXByaWwgMSksIEkgd2lsbCB0YWtlIG9uZSBzdGVwIGEgZGF5IHRvIGltcHJvdmUgbXkgZmluYW5jZXMgYW5kIGJsb2cgYWJvdXQgbXkgZXhwZXJpZW5jZS4gIiAvPmQCBRBkZBYCAgMPZBYCAgEPZBYCZg9kFgICAQ9kFgJmD2Q...
Upon inspection though, I confirmed that both CDs used the same MachineKey and so possibly the viewstate was corrupted or the churning CD had a problem validating it. In fact, it could be unrelated, but it was the only other sign we have to follow.
Based on the info we have so far, it was determined that we could or should use a SQL Session State server to avoid the DMS concurrency issues when this happens, preventing the endless retries and CPU getting maxed out. We planned to follow the Walkthrough: Configuring a shared session state database using the SQL Server provider
However, while doing research to do just that, I found information that made my question whether SQL Session State was even an option with our version of Sitecore:
- ASP.NET Session State support for Sitecore Client
- Sitecore DMS errors when using the ASP.NET out-of-process session-state modes
Ultimately the question is... What are our options to resolve this issue to properly configure load-balanced content delivery servers with DMS functionality? We need to be able to deploy OS patches, code updates, etc... so there is always the possibility that one server goes down or traffic is transferred to another node. Also, upgrading to a later version of Sitecore is not really an option at this point since a new site is being built on the recommended version to launch in the coming year and there is no desire to expend a lot of effort on the current solution.
- Version: Sitecore 6.5 Update-1 (rev. 120706)
- 1 Content Management server
- 2 load-balanced Content Delivery servers