I'm seeing a weird inconsistent problem that causes the Publishing Service to fail (we're on 2.0.0). Roughly once per day, usually in the morning, the Publish Jobs start failing. Doing an iisreset fixes the problem for a while, but that won't be acceptable on our Production environment.
I've seen this on our local dev computers, our dev server, and our production server. All of them have the Sitecore Publishing site running on the same machine as the Sitecore CM.
It's inconsistent, of course today I haven't been able to reproduce it locally so I don't have any screenshots, I'll post more information next time I see it happen.
The best lead I have is something I'm seeing in the logs:
2017-04-25 10:22:39.961 -04:00 [Fatal] Instance Activation strategy "MultiInstanceActivationStrategy" on "af7a5a55-de82-45a3-abad-4e196ca74604" failed to communicate with the ActivationLockRepository.
The Guid is different for the different environments, but it does not seem to be the id of an item in Sitecore master or core. I guess I can't be entirely sure that this message is related to our problem, but it seems very likely.
Here's the full log message:
2017-04-25 10:22:39.961 -04:00 [Fatal] Instance Activation strategy "MultiInstanceActivationStrategy" on "af7a5a55-de82-45a3-abad-4e196ca74604" failed to communicate with the ActivationLockRepository.
System.AggregateException: One or more errors occurred. ---> System.AggregateException: One or more errors occurred. ---> System.InvalidOperationException: Invalid operation. The connection is closed.
at System.Data.SqlClient.SqlConnection.GetOpenConnection()
at System.Data.SqlClient.SqlConnection.BeginTransaction(IsolationLevel iso, String transactionName)
at System.Data.SqlClient.SqlConnection.BeginDbTransaction(IsolationLevel isolationLevel)
at Sitecore.Framework.Publishing.Service.Sql.InstanceActivation.ActivationLockProvider.<>c__DisplayClass0_0.<<TryObtainLock>b__0>d.MoveNext() in C:\builds\1\s\src\Sitecore.Framework.Publishing.Service.Sql\InstanceActivation\ActivationLockProvider.cs:line 17
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Sitecore.Framework.Publishing.Data.AdoNet.NoRetryConnectionBehaviour.<ExecuteAsync>d__10`1.MoveNext() in C:\builds\1\s\src\Sitecore.Framework.Publishing.Data\AdoNet\NoRetryConnectionBehaviour.cs:line 50
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Sitecore.Framework.Publishing.Data.AdoNet.DatabaseConnection`1.<ExecuteAsync>d__27`1.MoveNext() in C:\builds\1\s\src\Sitecore.Framework.Publishing.Data\AdoNet\DatabaseConnection.cs:line 106
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
at Sitecore.Framework.Publishing.Service.Sql.InstanceActivation.ActivationLockProvider.<>c.<TryObtainLock>b__0_1(Task`1 x) in C:\builds\1\s\src\Sitecore.Framework.Publishing.Service.Sql\InstanceActivation\ActivationLockProvider.cs:line 40
at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
at System.Threading.Tasks.Task.Execute()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Sitecore.Framework.Publishing.Service.Sql.InstanceActivation.ActivationLockProvider.<TryObtainLock>d__0.MoveNext() in C:\builds\1\s\src\Sitecore.Framework.Publishing.Service.Sql\InstanceActivation\ActivationLockProvider.cs:line 15
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Sitecore.Framework.Publishing.InstanceActivation.ActivationLockRepository.<TryObtainLock>d__3.MoveNext() in C:\builds\1\s\src\Sitecore.Framework.Publishing\InstanceActivation\ActivationLockRepository.cs:line 28
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
at Sitecore.Framework.Publishing.InstanceActivation.MultiInstanceActivationStrategy.ObtainActivationLock(IActivationLockRepository repo) in C:\builds\1\s\src\Sitecore.Framework.Publishing\InstanceActivation\MultiInstanceActivationStrategy.cs:line 121
---> (Inner Exception #0) System.AggregateException: One or more errors occurred. ---> System.InvalidOperationException: Invalid operation. The connection is closed.
at System.Data.SqlClient.SqlConnection.GetOpenConnection()
at System.Data.SqlClient.SqlConnection.BeginTransaction(IsolationLevel iso, String transactionName)
at System.Data.SqlClient.SqlConnection.BeginDbTransaction(IsolationLevel isolationLevel)
at Sitecore.Framework.Publishing.Service.Sql.InstanceActivation.ActivationLockProvider.<>c__DisplayClass0_0.<<TryObtainLock>b__0>d.MoveNext() in C:\builds\1\s\src\Sitecore.Framework.Publishing.Service.Sql\InstanceActivation\ActivationLockProvider.cs:line 17
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Sitecore.Framework.Publishing.Data.AdoNet.NoRetryConnectionBehaviour.<ExecuteAsync>d__10`1.MoveNext() in C:\builds\1\s\src\Sitecore.Framework.Publishing.Data\AdoNet\NoRetryConnectionBehaviour.cs:line 50
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Sitecore.Framework.Publishing.Data.AdoNet.DatabaseConnection`1.<ExecuteAsync>d__27`1.MoveNext() in C:\builds\1\s\src\Sitecore.Framework.Publishing.Data\AdoNet\DatabaseConnection.cs:line 106
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
at Sitecore.Framework.Publishing.Service.Sql.InstanceActivation.ActivationLockProvider.<>c.<TryObtainLock>b__0_1(Task`1 x) in C:\builds\1\s\src\Sitecore.Framework.Publishing.Service.Sql\InstanceActivation\ActivationLockProvider.cs:line 40
at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
at System.Threading.Tasks.Task.Execute()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Sitecore.Framework.Publishing.Service.Sql.InstanceActivation.ActivationLockProvider.<TryObtainLock>d__0.MoveNext() in C:\builds\1\s\src\Sitecore.Framework.Publishing.Service.Sql\InstanceActivation\ActivationLockProvider.cs:line 15
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Sitecore.Framework.Publishing.InstanceActivation.ActivationLockRepository.<TryObtainLock>d__3.MoveNext() in C:\builds\1\s\src\Sitecore.Framework.Publishing\InstanceActivation\ActivationLockRepository.cs:line 28
---> (Inner Exception #0) System.InvalidOperationException: Invalid operation. The connection is closed.
at System.Data.SqlClient.SqlConnection.GetOpenConnection()
at System.Data.SqlClient.SqlConnection.BeginTransaction(IsolationLevel iso, String transactionName)
at System.Data.SqlClient.SqlConnection.BeginDbTransaction(IsolationLevel isolationLevel)
at Sitecore.Framework.Publishing.Service.Sql.InstanceActivation.ActivationLockProvider.<>c__DisplayClass0_0.<<TryObtainLock>b__0>d.MoveNext() in C:\builds\1\s\src\Sitecore.Framework.Publishing.Service.Sql\InstanceActivation\ActivationLockProvider.cs:line 17
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Sitecore.Framework.Publishing.Data.AdoNet.NoRetryConnectionBehaviour.<ExecuteAsync>d__10`1.MoveNext() in C:\builds\1\s\src\Sitecore.Framework.Publishing.Data\AdoNet\NoRetryConnectionBehaviour.cs:line 50
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Sitecore.Framework.Publishing.Data.AdoNet.DatabaseConnection`1.<ExecuteAsync>d__27`1.MoveNext() in C:\builds\1\s\src\Sitecore.Framework.Publishing.Data\AdoNet\DatabaseConnection.cs:line 106<---
<---