9

Our problem is that periodically one of the CD servers CPU soars to 100% and IIS needs to be restarted to get it to calm down. While investigating the logs we found the following two stacktraces:

ManagedPoolThread #97 15:45:40 ERROR Error committing data row Pages.ef42d298-9973-4c98-9457-1ad4c4761749 : Concurrency violation: the UpdateCommand affected 0 of the expected 1 records.
ManagedPoolThread #97 15:45:40 WARN  Errors while committing data. Retrying...
System.Exception: Error committing datatable: Pages ---> System.Exception: Commit error: Concurrency violation: the UpdateCommand affected 0 of the expected 1 records.
   at Sitecore.Analytics.Data.DataAccess.DataAdapters.Sql.SqlDataAdapterProvider.Update(DataTable table, UpdateOptions options)
   at Sitecore.Analytics.Data.DataAccess.DataAdapters.Sql.SqlDataAdapterProvider.Update(DataTable table)
   at Sitecore.Analytics.Data.DataAccess.DataAdapters.Sql.SqlDataAdapterProvider.Update(DataSet dataSet)
   --- End of inner exception stack trace ---
   at Sitecore.Pipelines.CorePipeline.Run(PipelineArgs args)
   at Sitecore.Pipelines.CorePipeline.Run(String pipelineName, PipelineArgs args, String pipelineDomain, Boolean failIfNotExists)
   at Sitecore.Pipelines.CorePipeline.Run(String pipelineName, PipelineArgs args, String pipelineDomain)
   at Sitecore.Pipelines.CorePipeline.Run(String pipelineName, PipelineArgs args)
   at Sitecore.Analytics.Pipelines.CommitDataSet.CommitDataSetPipeline.Run(CommitDataSetArgs args)
   at Sitecore.Analytics.Data.TrackerChanges.__AW_Flush()

For this particular issue we found this knowledge base article (Concurrency violation issues in DMS) which indicated that perhaps a visitor had bounced from one of the load-balanced CDs to the other. The load balancer is configured for sticky sessions but the visitor could have transferred for several reasons like one of the cookies expired, one of the CDs was being taken out of the pool temporarily, etc. Since our scenario lined up with "Scenario 1" in the documentation (Load balancer with 2 or more CDs, and InProc Session State mode), our solution could be to configure a SQLServer based session state.

Another error we saw at the same time on the same CD was one that is pretty familiar when the CDs aren't configured properly or the load balancer isn't using sticky sessions:

34320 21:16:21 ERROR Application error.
Exception: System.Web.HttpException
Message: Validation of viewstate MAC failed. If this application is hosted by a Web Farm or cluster, ensure that <machineKey> configuration specifies the same validationKey and validation algorithm. AutoGenerate cannot be used in a cluster.

See http://go.microsoft.com/fwlink/?LinkID=314055 for more information.
Source: System.Web
   at System.Web.UI.ViewStateException.ThrowError(Exception inner, String persistedState, String errorPageMessage, Boolean macValidationError)
   at System.Web.UI.ViewStateException.ThrowMacValidationError(Exception inner, String persistedState)
   at System.Web.UI.ObjectStateFormatter.Deserialize(String inputString, Purpose purpose)
   at System.Web.UI.ObjectStateFormatter.System.Web.UI.IStateFormatter2.Deserialize(String serializedState, Purpose purpose)
   at System.Web.UI.Util.DeserializeWithAssert(IStateFormatter2 formatter, String serializedState, Purpose purpose)
   at System.Web.UI.HiddenFieldPageStatePersister.Load()
   at System.Web.UI.Page.LoadPageStateFromPersistenceMedium()
   at System.Web.UI.Page.LoadAllState()
   at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)
   at System.Web.UI.Page.ProcessRequest(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)
   at System.Web.UI.Page.ProcessRequest()
   at System.Web.UI.Page.ProcessRequestWithNoAssert(HttpContext context)
   at System.Web.UI.Page.ProcessRequest(HttpContext context)
   at ASP._layouts_flm_masterpages_baselayout_aspx.__AW_ProcessRequest(HttpContext )
   at ASP._layouts_flm_masterpages_baselayout_aspx.ProcessRequest(HttpContext context)
   at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
   at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)

Nested Exception

Exception: System.Web.UI.ViewStateException
Message: Invalid viewstate. 
    Client IP: 10.8.5.172
    Port: 44240
    Referer: http://xxxxxxxxxxxxxxxxx/Blog/Default.aspx
    Path: /Blog/Default.aspx
    User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.109 Safari/537.36
    ViewState: /wEPDwUKLTkxNzg0OTU0NQ9kFgQCAQ9kFgJmD2QWBgIDDxYCHgRUZXh0BRsgLSBGaW5hbmNpYWwgTGl0ZXJhY3kgTW9udGhkAgUPFgIfAAXFATxtZXRhIG5hbWU9ImtleXdvcmRzIiBjb250ZW50PSJGaW5hbmNpYWwgYmxvZ3MsIGZpbmFuY2lhbCBibG9nLCBmaW5hbmNpYWwgbGl0ZXJhY3kgYmxvZywgZmluYW5jaWFsIHNlcnZpY2VzIGJsb2csIGZpbmFuY2lhbCBzdWNjZXNzIGJsb2csLCBwZXJzb25hbCBidWRnZXRpbmcgYmxvZywgZmluYW5jaWFsIGxpdGVyYWN5IG1vbnRoIGJsb2ciIC8+ZAIHDxYCHwAF7wE8bWV0YSBuYW1lPSJkZXNjcmlwdGlvbiIgY29udGVudD0iRmluYW5jaWFsIExpdGVyYWN5IE1vbnRoIEJsb2cgYWJvdXQgRmluYW5jaWFsIFN1Y2Nlc3MuICBTdGFydGluZyBvbiB0aGUgZmlyc3QgZGF5IG9mIEZpbmFuY2lhbCBMaXRlcmFjeSBNb250aCAoQXByaWwgMSksIEkgd2lsbCB0YWtlIG9uZSBzdGVwIGEgZGF5IHRvIGltcHJvdmUgbXkgZmluYW5jZXMgYW5kIGJsb2cgYWJvdXQgbXkgZXhwZXJpZW5jZS4gIiAvPmQCBRBkZBYCAgMPZBYCAgEPZBYCZg9kFgICAQ9kFgJmD2Q...

Upon inspection though, I confirmed that both CDs used the same MachineKey and so possibly the viewstate was corrupted or the churning CD had a problem validating it. In fact, it could be unrelated, but it was the only other sign we have to follow.

Based on the info we have so far, it was determined that we could or should use a SQL Session State server to avoid the DMS concurrency issues when this happens, preventing the endless retries and CPU getting maxed out. We planned to follow the Walkthrough: Configuring a shared session state database using the SQL Server provider

However, while doing research to do just that, I found information that made my question whether SQL Session State was even an option with our version of Sitecore:

Ultimately the question is... What are our options to resolve this issue to properly configure load-balanced content delivery servers with DMS functionality? We need to be able to deploy OS patches, code updates, etc... so there is always the possibility that one server goes down or traffic is transferred to another node. Also, upgrading to a later version of Sitecore is not really an option at this point since a new site is being built on the recommended version to launch in the coming year and there is no desire to expend a lot of effort on the current solution.

Configuration:

  • Version: Sitecore 6.5 Update-1 (rev. 120706)
  • 1 Content Management server
  • 2 load-balanced Content Delivery servers
Improve this question
2
  • 1
    Perhaps not really a solution... but I've checked out the release notes of Sitecore 6.5 update 6 (Service Pack 2) which includes a fix for the error you are describing: The log file would fill up with errors messages stating “Some of the automation states could not be written to the database, probably because of expired lock. Record IDs below: <guid> Concurrency violation: the UpdateCommand affected 0 of the expected 1 records.” (372193). (also keep in mind that the support for 6.5 will end in 2019: kb.sitecore.net/articles/641167)
    – Koen Heye
    Oct 12, 2016 at 18:19
  • 1
    FYI As I recall the 2 referenced links relate to the Sitecore client. Since you only have a single CM server, there's no reason why that shouldn't remain InProc.
    – Martin Davies
    Oct 12, 2016 at 18:27

1 Answer 1

Reset to default
3

For a 6.5 update 1 using DMS you are unfortunately out of luck. In 6.5, you cannot load balance the CM servers (you dont need it but just stated for completeness) In 6.5 versions before update 4 you cannot use a shared session for DMS.

You cannot do much else than recheck the loadbalancer configuration for stickyness.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.