8

I'm seeing a weird inconsistent problem that causes the Publishing Service to fail (we're on 2.0.0). Roughly once per day, usually in the morning, the Publish Jobs start failing. Doing an iisreset fixes the problem for a while, but that won't be acceptable on our Production environment.

I've seen this on our local dev computers, our dev server, and our production server. All of them have the Sitecore Publishing site running on the same machine as the Sitecore CM.

It's inconsistent, of course today I haven't been able to reproduce it locally so I don't have any screenshots, I'll post more information next time I see it happen.

The best lead I have is something I'm seeing in the logs:

2017-04-25 10:22:39.961 -04:00 [Fatal] Instance Activation strategy "MultiInstanceActivationStrategy" on "af7a5a55-de82-45a3-abad-4e196ca74604" failed to communicate with the ActivationLockRepository.

The Guid is different for the different environments, but it does not seem to be the id of an item in Sitecore master or core. I guess I can't be entirely sure that this message is related to our problem, but it seems very likely.

Here's the full log message:

2017-04-25 10:22:39.961 -04:00 [Fatal] Instance Activation strategy "MultiInstanceActivationStrategy" on "af7a5a55-de82-45a3-abad-4e196ca74604" failed to communicate with the ActivationLockRepository.
System.AggregateException: One or more errors occurred. ---> System.AggregateException: One or more errors occurred. ---> System.InvalidOperationException: Invalid operation. The connection is closed.
   at System.Data.SqlClient.SqlConnection.GetOpenConnection()
   at System.Data.SqlClient.SqlConnection.BeginTransaction(IsolationLevel iso, String transactionName)
   at System.Data.SqlClient.SqlConnection.BeginDbTransaction(IsolationLevel isolationLevel)
   at Sitecore.Framework.Publishing.Service.Sql.InstanceActivation.ActivationLockProvider.<>c__DisplayClass0_0.<<TryObtainLock>b__0>d.MoveNext() in C:\builds\1\s\src\Sitecore.Framework.Publishing.Service.Sql\InstanceActivation\ActivationLockProvider.cs:line 17
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Sitecore.Framework.Publishing.Data.AdoNet.NoRetryConnectionBehaviour.<ExecuteAsync>d__10`1.MoveNext() in C:\builds\1\s\src\Sitecore.Framework.Publishing.Data\AdoNet\NoRetryConnectionBehaviour.cs:line 50
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Sitecore.Framework.Publishing.Data.AdoNet.DatabaseConnection`1.<ExecuteAsync>d__27`1.MoveNext() in C:\builds\1\s\src\Sitecore.Framework.Publishing.Data\AdoNet\DatabaseConnection.cs:line 106
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at Sitecore.Framework.Publishing.Service.Sql.InstanceActivation.ActivationLockProvider.<>c.<TryObtainLock>b__0_1(Task`1 x) in C:\builds\1\s\src\Sitecore.Framework.Publishing.Service.Sql\InstanceActivation\ActivationLockProvider.cs:line 40
   at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
   at System.Threading.Tasks.Task.Execute()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Sitecore.Framework.Publishing.Service.Sql.InstanceActivation.ActivationLockProvider.<TryObtainLock>d__0.MoveNext() in C:\builds\1\s\src\Sitecore.Framework.Publishing.Service.Sql\InstanceActivation\ActivationLockProvider.cs:line 15
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Sitecore.Framework.Publishing.InstanceActivation.ActivationLockRepository.<TryObtainLock>d__3.MoveNext() in C:\builds\1\s\src\Sitecore.Framework.Publishing\InstanceActivation\ActivationLockRepository.cs:line 28
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at Sitecore.Framework.Publishing.InstanceActivation.MultiInstanceActivationStrategy.ObtainActivationLock(IActivationLockRepository repo) in C:\builds\1\s\src\Sitecore.Framework.Publishing\InstanceActivation\MultiInstanceActivationStrategy.cs:line 121
---> (Inner Exception #0) System.AggregateException: One or more errors occurred. ---> System.InvalidOperationException: Invalid operation. The connection is closed.
   at System.Data.SqlClient.SqlConnection.GetOpenConnection()
   at System.Data.SqlClient.SqlConnection.BeginTransaction(IsolationLevel iso, String transactionName)
   at System.Data.SqlClient.SqlConnection.BeginDbTransaction(IsolationLevel isolationLevel)
   at Sitecore.Framework.Publishing.Service.Sql.InstanceActivation.ActivationLockProvider.<>c__DisplayClass0_0.<<TryObtainLock>b__0>d.MoveNext() in C:\builds\1\s\src\Sitecore.Framework.Publishing.Service.Sql\InstanceActivation\ActivationLockProvider.cs:line 17
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Sitecore.Framework.Publishing.Data.AdoNet.NoRetryConnectionBehaviour.<ExecuteAsync>d__10`1.MoveNext() in C:\builds\1\s\src\Sitecore.Framework.Publishing.Data\AdoNet\NoRetryConnectionBehaviour.cs:line 50
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Sitecore.Framework.Publishing.Data.AdoNet.DatabaseConnection`1.<ExecuteAsync>d__27`1.MoveNext() in C:\builds\1\s\src\Sitecore.Framework.Publishing.Data\AdoNet\DatabaseConnection.cs:line 106
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at Sitecore.Framework.Publishing.Service.Sql.InstanceActivation.ActivationLockProvider.<>c.<TryObtainLock>b__0_1(Task`1 x) in C:\builds\1\s\src\Sitecore.Framework.Publishing.Service.Sql\InstanceActivation\ActivationLockProvider.cs:line 40
   at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
   at System.Threading.Tasks.Task.Execute()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Sitecore.Framework.Publishing.Service.Sql.InstanceActivation.ActivationLockProvider.<TryObtainLock>d__0.MoveNext() in C:\builds\1\s\src\Sitecore.Framework.Publishing.Service.Sql\InstanceActivation\ActivationLockProvider.cs:line 15
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Sitecore.Framework.Publishing.InstanceActivation.ActivationLockRepository.<TryObtainLock>d__3.MoveNext() in C:\builds\1\s\src\Sitecore.Framework.Publishing\InstanceActivation\ActivationLockRepository.cs:line 28
---> (Inner Exception #0) System.InvalidOperationException: Invalid operation. The connection is closed.
   at System.Data.SqlClient.SqlConnection.GetOpenConnection()
   at System.Data.SqlClient.SqlConnection.BeginTransaction(IsolationLevel iso, String transactionName)
   at System.Data.SqlClient.SqlConnection.BeginDbTransaction(IsolationLevel isolationLevel)
   at Sitecore.Framework.Publishing.Service.Sql.InstanceActivation.ActivationLockProvider.<>c__DisplayClass0_0.<<TryObtainLock>b__0>d.MoveNext() in C:\builds\1\s\src\Sitecore.Framework.Publishing.Service.Sql\InstanceActivation\ActivationLockProvider.cs:line 17
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Sitecore.Framework.Publishing.Data.AdoNet.NoRetryConnectionBehaviour.<ExecuteAsync>d__10`1.MoveNext() in C:\builds\1\s\src\Sitecore.Framework.Publishing.Data\AdoNet\NoRetryConnectionBehaviour.cs:line 50
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Sitecore.Framework.Publishing.Data.AdoNet.DatabaseConnection`1.<ExecuteAsync>d__27`1.MoveNext() in C:\builds\1\s\src\Sitecore.Framework.Publishing.Data\AdoNet\DatabaseConnection.cs:line 106<---
<---
Improve this question

1 Answer 1

Reset to default
3

Edit 14/6/17: This has indeed been fixed in v2.0 Update 1 (release notes)

The error message you are seeing is indeed the cause of the Publish Service ceasing to function.

Basically, the Service is configured to always assume it's running in a multi-instance setup. On a repeating interval, it will try to obtain an 'activation lock', i.e. it will try and nominate itself as the leader instance. If it fails (i.e. there is already another leader or the DB is down), then the service will become dormant.

In fact, you are encountering an unforeseen scenario (which we have never seen before), hence the exception propagation - something is causing the underlying connection to be closed during the lifetime of this activation algorithm. The algorithm should be tolerant of this - it's a bug that we will aim to include in the next release.

For now, if you are only running the Publishing Service in a single instance setup, you can switch to the simpler 'single instance activation strategy' that is shipped with the service, but is not enabled by default.

The following Service configuration will set the activation strategy to the Single instance implementation. This will always set the Service to be the leader, and doesn't require any DB communication. Please see the Service documentation for details on how to provide custom configuration.

<Settings>
  <Sitecore>
    <Publishing>
      <Services>
        <InstanceActivationStrategy>
           <Type>Sitecore.Framework.Publishing.InstanceActivation.SingleInstanceActivationStrategy, Sitecore.Framework.Publishing</Type>
        </InstanceActivationStrategy>
      </Services> 
    </Publishing>
  </Sitecore>
</Settings>
Improve this answer
3
  • Awesome, this looks promising. I'll give it a shot and accept the answer in a few days if the problem goes away. Thanks for your help.
    – Slee
    Apr 27, 2017 at 15:52
  • A few of us hit this in a training session today while the core db was getting pinned for some reason. This change fixed it!
    – Jason St-Cyr
    Apr 28, 2017 at 22:23
  • Changed the strategy and recycled the apppool on the service. Seems to be back up and running for me.
    – Michael West
    Jun 20, 2017 at 19:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.