Custom Azure Index doesn't update on CD Servers after rebuild

Question

We have a custom Azure Index which we have enabled switch on rebuild for.

The works fine on the CM servers but. However on CD we get no data showing until we restart the Azure App Service for the CD.

I'm not sure what might be causing this but it seems like switch on rebuild is firing fine on the CM server and then it's deleting the old index. However on CD it is still looking at the old index as has not been told to look at the new index.

I have installed the patch from Sitecore for a known issue with this on 8.2. So this shouldn't be an issue anymore.

The config which is present on both CD and CM for our custom index includes the following. Should this be on both CD and CM or only on CM?:

...
<pipelines>
  <initialize>
    <processor type="Sitecore.ContentSearch.Azure.Events.RebuildEvents.SwitchOnRebuildEventHandler, Sitecore.ContentSearch.Azure" method="InitializeFromPipeline" />
  </initialize>
</pipelines>
<events>
  <event name="index:switchonrebuild">
    <handler type="Sitecore.ContentSearch.Azure.Events.RebuildEvents.SwitchOnRebuildEventHandler, Sitecore.ContentSearch.Azure" method="OnSwitchOnRebuildEvent" />
  </event>
  <event name="index:switchonrebuild:remote">
    <handler type="Sitecore.ContentSearch.Azure.Events.RebuildEvents.SwitchOnRebuildEventHandler, Sitecore.ContentSearch.Azure" method="OnSwitchOnRebuildEvent" />
  </event>
</events>

This is the config for the custom search index:

 <contentSearch>
  <configuration type="Sitecore.ContentSearch.ContentSearchConfiguration, Sitecore.ContentSearch">
    <indexes hint="list:AddIndex">
      <index id="my-custom-index" type="Sitecore.ContentSearch.Azure.CloudSearchProviderIndex, Sitecore.ContentSearch.Azure">
        <param desc="name">$(id)</param>
        <param desc="connectionStringName">cloud.search</param>
        <param desc="totalParallelServices">5</param>
        <!-- This initializes index property store. Id has to be set to the index id -->
        <param desc="propertyStore" ref="contentSearch/indexConfigurations/databasePropertyStore" param1="$(id)" />
        <configuration ref="contentSearch/indexConfigurations/customCloudIndexConfiguration" />
        <schemaBuilder ref="contentSearch/searchServiceSchemaBuilder" />
        <searchService type="Sitecore.ContentSearch.Azure.Http.CompositeSearchService, Sitecore.ContentSearch.Azure"/>
        <strategies hint="list:AddStrategy">
          <strategy ref="contentSearch/indexConfigurations/indexUpdateStrategies/manual" />
        </strategies>
        <locations hint="list:AddCrawler">
          <crawler type="MyCustom.ContentSearch.MySearchCrawler, MyCustom">
          </crawler>  
        </locations>
      </index>
    </indexes>
  </configuration>
</contentSearch>

using: Sitecore 8.2 u7