Remote powershell scripts failing intermittently

Question

During deployment with AzureDevOps, remote PowerShell scripts fail intermittently.

In SPE logs we see,

60796 08:22:06 INFO 'POST/script' called by user: 'admin'
61320 08:22:09 INFO Executing PsSitecoreItemProvider5.Start(providerInfo='Sitecore')
60796 08:22:11 INFO Executing a script in ScriptSession '$scriptSession$|gin1sbn1qxwau3esknesj4lo|6bf5df1e-fd17-49ea-be77-0d5781f06f4d'.
60796 08:22:11 ERROR 'ScriptSessionNotFound' (Category: ResourceUnavailable) error encountered on object.
Exception: System.Data.ObjectNotFoundException
Message: The script session with Id 'de3c7ec8-59ce-40c6-9b33-ed950c57a8bd' cannot be found.

60796 08:22:11 INFO Executing a script in ScriptSession '$scriptSession$|gin1sbn1qxwau3esknesj4lo|6bf5df1e-fd17-49ea-be77-0d5781f06f4d'.
28032 08:22:12 INFO 'POST/script' called by user: 'admin'
56168 08:22:12 INFO Executing PsSitecoreItemProvider5.Start(providerInfo='Sitecore')
28032 08:22:12 INFO Executing a script in ScriptSession '$scriptSession$|rawqu0ozcfhhbkj3rhe1ihos|6bf5df1e-fd17-49ea-be77-0d5781f06f4d'.
28032 08:22:12 ERROR 'ScriptSessionNotFound' (Category: ResourceUnavailable) error encountered on object.
Exception: System.Data.ObjectNotFoundException
Message: The script session with Id 'de3c7ec8-59ce-40c6-9b33-ed950c57a8bd' cannot be found.

In Azure DevOps we see,

2021-10-08T05:16:08.6791082Z ##[error]The script session with Id '8723e50b-7b0b-4ae4-ba56-63cfeb6fe3d6' cannot be received because it is in the Busy state. Use Stop-ScriptSession or wait for the operation to complete.
2021-10-08T05:16:08.7869877Z ##[error]PowerShell exited with code '1'.

2021-10-06T03:42:26.2769961Z No jobId was created. Please check if your Powershell Remoting is activated on the target instance 'https://xxx.xxx.xxx'
2021-10-06T03:42:26.2772950Z ##[error]Cannot validate argument on parameter 'Id'. The argument is null or empty. Provide an argument that is not null or empty, and then try the command again.
2021-10-06T03:42:26.2776171Z ##[error]PowerShell exited with code '1'.

We use the following script to execute remote powershell script,

param (
[Parameter(Mandatory = $true, HelpMessage = 'Sitecore username to run script as')]
[string]$username,
[Parameter(Mandatory = $true, HelpMessage = 'Sitecore password')]
[SecureString]$password,
[Parameter(Mandatory = $true, HelpMessage = 'URL to the Sitecore instance')]
[string]$connectionUrl,
[Parameter(Mandatory = $true, HelpMessage = 'Script to run remotely')]
[scriptblock]$scriptBlock,
[Parameter(HelpMessage = 'The polling interval in seconds.')]
[int]$pollingInterval=30,
[Parameter(HelpMessage = 'Sleep interval between invoking remote script and polling state of remote script.')]
[int]$sleepInterval=10
)

Write-Host
Write-Host
Write-Host ===================================================================================================== -ForegroundColor Magenta
Write-Host ===================================================================================================== -ForegroundColor Magenta
Write-Host Run webjob script -ForegroundColor Magenta
Write-Host
foreach ($key in $MyInvocation.BoundParameters.keys) {
if (!$key.ToLower().Contains("password")) {
Write-Host " Parameter: $($key) -> $($MyInvocation.BoundParameters[$key])"
}
}
Write-Host

Install-Module SPE -Scope CurrentUser -AllowClobber -Force
Import-Module SPE

$BSTR = [System.Runtime.InteropServices.Marshal]::SecureStringToBSTR($password)
$p = [System.Runtime.InteropServices.Marshal]::PtrToStringAuto($BSTR)

$session = New-ScriptSession -Username $username -Password $p -ConnectionUri $connectionUrl
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
$jobId = Invoke-RemoteScript -Session $session -ScriptBlock $scriptBlock -Raw -AsJob -Verbose

if (!$jobId) {
Write-Host "No jobId was created. Please check if your Powershell Remoting is activated on the target instance '$connectionUrl'"
}

<# Sleeping for 10secs based on the SPE module example, otherwise the Wait-RemoteScriptSession seems to be returning following error intermittently
#65984 11:08:07 ERROR 'ScriptSessionNotFound' (Category: ResourceUnavailable) error encountered on object.
#Exception: System.Data.ObjectNotFoundException
Message: The script session with Id '0b49cbde-a2fb-4ad4-b26e-fec4d9215103' cannot be found.
#>
Start-Sleep -Seconds $sleepInterval

$ss = Wait-RemoteScriptSession -Session $session -Id $jobId -Delay $pollingInterval -Raw -Verbose

# Always stopping script session at the end of Wait-RemoteScriptSession to avoid any issues with the following SPE scripts
Stop-ScriptSession -Session $session

if($ss.LastErrors) {
Write-Host "Errors from remote session"
$ss.LastErrors
throw $ss.LastErrors
}