Could SocketException be caused by CloudSearchRetryPolicy?

Question

I'm investigating an issue where we are seeing SocketExceptions and socket handle leaks. Using Sitecore 8.2 rev 107614, in Azure using Azure Search as the search provider.

Drilling down to one of the exceptions, I see the message

An unhandled exception occurred. Error while rendering view: '/Views/Layouts/Layout.cshtml' (model: 'Sitecore.Mvc.Presentation.RenderingModel, Sitecore.Mvc'). An unhandled exception occurred. An unhandled exception occurred. An error occurred while sending the request. Unable to connect to the remote server An attempt was made to access a socket in a way forbidden by its access permissions

The exception type is

System.Net.Sockets.SocketException

and the Failed method is

Sitecore.ContentSearch.Azure.Http.RetryDelegatingHandler+<>c__DisplayClass2_0.b__0

So I'm wondering if this is related to how the search service client is configured as I see there are some re-tryer configuration settings (left at default, I believe)

<searchServiceClient type="Sitecore.ContentSearch.Azure.Http.SearchServiceClient, Sitecore.ContentSearch.Azure">
  <param ref="contentSearch/httpClientFactory" desc="clientFactory"/>
  <param type="Sitecore.ContentSearch.Azure.Http.CloudSearchRetryPolicy, Sitecore.ContentSearch.Azure" desc="retryPolicy">
    <DetectionStrategy type="Sitecore.ContentSearch.Azure.Http.CloudSearchTransientDetectionStrategy, Sitecore.ContentSearch.Azure"/>
    <RetryStrategy type="Sitecore.ContentSearch.Azure.Http.CloudSearchExponentialRetryStrategy, Sitecore.ContentSearch.Azure">
      <Exponent>1.5</Exponent>
      <MaxRetryCount>10</MaxRetryCount>
      <MinDelayInterval>00:00:00.5</MinDelayInterval>
      <MaxDelayInterval>00:00:10</MaxDelayInterval>
    </RetryStrategy>
  </param>
  <managementOperationsTimeout>00:01:00</managementOperationsTimeout>
</searchServiceClient>

Question(s)

When does the "re-tryer" come into play, could be it during unavailability of the search service? Is there any documentation about the re-tryer, or does someone know how I might modify these settings to avoid the SocketException issue I am seeing? (I can guess what MaxReTryCount, MinDelayInterval and MaxDelayInterval do, but not sure what the Exponent property is for?).

Answer 1

Yes, you are on the right track. If you are getting issues with connecting / submitting to Azure Search then it's possible you are hitting the limit of 10 "GetIndex" requests per second.

My understanding is that if too many Sitecore instances are doing "GetIndex" at the same time, then it will trigger the Azure Search limit. The defaults are not forgiving in this case as they frequently and exponentially will retry.

Tweaking your delays as follows will help a lot. (This is a Sitecore 9.0.2 config - perhaps just take the relevant parts for your version).

  <schemaSynchronizer type="Sitecore.ContentSearch.Azure.Schema.SearchServiceSchemaSynchronizer, Sitecore.ContentSearch.Azure">
    <param desc="managmentOperations" ref="contentSearch/searchServiceClient" />
    <param desc="retryPolicy" type="Sitecore.ContentSearch.Azure.Utils.Retryer.RetryPolicy, Sitecore.ContentSearch.Azure">
      <param desc="retryStrategy" type="Sitecore.ContentSearch.Azure.Utils.Retryer.ExponentialRetryStrategy, Sitecore.ContentSearch.Azure">
        <exponent>1.2</exponent>
        <maxRetryCount>10</maxRetryCount>
        <minDelayInterval>00:00:10.0</minDelayInterval>
        <maxDelayInterval>00:01:00.0</maxDelayInterval>
      </param>
      <param desc="detectionStrategy" type="Sitecore.ContentSearch.Azure.Utils.Retryer.SchemaUpdateErrorTransientDetectionStrategy, Sitecore.ContentSearch.Azure" />
    </param>
    <param desc="analyzerRepository" ref="contentSearch/analyzerRepository" />
  </schemaSynchronizer>

Additionally set the <param desc="schemaUpdateInterval">00:05:00</param>