Skip to main content
Sitecore

Return to Answer

Updated for 9.1
Source Link
Mark Gibbons
Mark Gibbons
  • 5.7k
  • 1
  • 19
  • 61
<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
  <sitecore>
    <settings>
      <!-- Enables indexing of cache keys as an experimental optimization to speed up publishing. -->
      <setting name="Caching.CacheKeyIndexingEnabled.AccessResultCache">
        <patch:attribute name="value">true</patch:attribute>
      </setting>
      <!-- Enables indexing of cache keys as an experimental optimization to speed up publishing. -->
      <setting name="Caching.CacheKeyIndexingEnabled.ItemCache">
        <patch:attribute name="value">true</patch:attribute>
      </setting>
      <!-- Enables indexing of cache keys as an experimental optimization to speed up publishing. -->
      <setting name="Caching.CacheKeyIndexingEnabled.ItemPathsCache">
        <patch:attribute name="value">true</patch:attribute>
      </setting>
      <!-- Disables performance counters as an experimental optimization to speed up publishing.
           You should comment out this element if you need to use performance counters. -->
      <setting name="Counters.Enabled">
        <patch:attribute name="value">false</patch:attribute>
      </setting>
      <!-- Disables incremental updates to the link database during publishing operations as an experimental optimization to speed up
           publishing.
           You should comment out this element if you use the LinkDatabase API on your content delivery instances. -->
      <setting name="LinkDatabase.UpdateDuringPublish">
        <patch:attribute name="value">false</patch:attribute>
      </setting>
      <!-- Disables rule item event handlers as an experimental optimization to speed up publishing.
           This setting only affects the instance that runs the publishing operation.
           You should comment out this element if there are rules in any of the following folders that must be executed during publishing:
           /sitecore/system/Settings/Rules/Item Saved/Rules
           /sitecore/system/Settings/Rules/Item Deleted/Rules
           /sitecore/system/Settings/Rules/Version Removed/Rules -->
      <setting name="Rules.ItemEventHandlers.RunDuringPublishing">
        <patch:attribute name="value">false</patch:attribute>
      </setting>
      <!-- This setting is enabled as an experimental optimization to speed up publishing.
           The following configuration limits the execution of rules by item event handlers in the core and master databases.
           You should comment out this element if there are rules in any of the following folders that must be executed on the web database
           or on any custom databases:
           /sitecore/system/Settings/Rules/Item Saved/Rules
           /sitecore/system/Settings/Rules/Item Deleted/Rules
           /sitecore/system/Settings/Rules/Version Removed/Rules -->
      <setting name="Rules.ItemEventHandlers.SupportedDatabases">
        <patch:attribute name="value">core|master</patch:attribute>
      </setting>
      <!-- This setting is changed from the default value to enable parallel processing of publishing candidates.
           The value '4 ' is specified here as an example.
           The optimal value for this setting depends on your solution and on the CPU capacity of the server that runs the publishing operations.

           For example, if the system architecture leverages a dedicated publishing instance with plenty of CPU capacity, this setting can be
           increased even further.
           On the other hand, on solutions where publishing is run on an authoring server that is already running a high load, changing this
           setting to a non-default value may cause system stability issues. 
           We highly recommend that you perform publishing tests with the expected load before you change this setting in production environments.

           Please remember that your database server's CPU and disk capacity may need to be adjusted to accommodate this setting.
      -->
      <setting name="Publishing.MaxDegreeOfParallelism">
        <patch:attribute name="value">4</patch:attribute>
      </setting>
    </settings>
    <pipelines>
      <publish>
        <!-- This processor performs parallel processing of publishing candidates.
             The Publishing.MaxDegreeOfParallelism setting specifies how many concurrent operations can be run. -->
        <processor type="Sitecore.Publishing.Pipelines.Publish.ProcessQueue, Sitecore.Kernel">
          <patch:attribute name="type">Sitecore.Publishing.Pipelines.ParallelPublish.ParallelProcessQueue,Sitecore.Kernel</patch:attribute>
        </processor>
      </publish>
    </pipelines>
  </sitecore>
</configuration>

From my calculation you are currently getting ~2 items/s which is in line with what I was getting before adding the config. If you can achieve ~100 items/s then you should be able to publish 200,000 items in ~34 minutes.

Edit

I've updated the config and removed a lot of the settings as I did have some issues with it on 9.1. Also the main setting to tweak is the Publishing.MaxDegreeOfParallelism as the others didn't have much effect from my testing.

<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
  <sitecore>
    <settings>
      <!-- Enables indexing of cache keys as an experimental optimization to speed up publishing. -->
      <setting name="Caching.CacheKeyIndexingEnabled.AccessResultCache">
        <patch:attribute name="value">true</patch:attribute>
      </setting>
      <!-- Enables indexing of cache keys as an experimental optimization to speed up publishing. -->
      <setting name="Caching.CacheKeyIndexingEnabled.ItemCache">
        <patch:attribute name="value">true</patch:attribute>
      </setting>
      <!-- Enables indexing of cache keys as an experimental optimization to speed up publishing. -->
      <setting name="Caching.CacheKeyIndexingEnabled.ItemPathsCache">
        <patch:attribute name="value">true</patch:attribute>
      </setting>
      <!-- Disables performance counters as an experimental optimization to speed up publishing.
           You should comment out this element if you need to use performance counters. -->
      <setting name="Counters.Enabled">
        <patch:attribute name="value">false</patch:attribute>
      </setting>
      <!-- Disables incremental updates to the link database during publishing operations as an experimental optimization to speed up
           publishing.
           You should comment out this element if you use the LinkDatabase API on your content delivery instances. -->
      <setting name="LinkDatabase.UpdateDuringPublish">
        <patch:attribute name="value">false</patch:attribute>
      </setting>
      <!-- Disables rule item event handlers as an experimental optimization to speed up publishing.
           This setting only affects the instance that runs the publishing operation.
           You should comment out this element if there are rules in any of the following folders that must be executed during publishing:
           /sitecore/system/Settings/Rules/Item Saved/Rules
           /sitecore/system/Settings/Rules/Item Deleted/Rules
           /sitecore/system/Settings/Rules/Version Removed/Rules -->
      <setting name="Rules.ItemEventHandlers.RunDuringPublishing">
        <patch:attribute name="value">false</patch:attribute>
      </setting>
      <!-- This setting is enabled as an experimental optimization to speed up publishing.
           The following configuration limits the execution of rules by item event handlers in the core and master databases.
           You should comment out this element if there are rules in any of the following folders that must be executed on the web database
           or on any custom databases:
           /sitecore/system/Settings/Rules/Item Saved/Rules
           /sitecore/system/Settings/Rules/Item Deleted/Rules
           /sitecore/system/Settings/Rules/Version Removed/Rules -->
      <setting name="Rules.ItemEventHandlers.SupportedDatabases">
        <patch:attribute name="value">core|master</patch:attribute>
      </setting>
      <!-- This setting is changed from the default value to enable parallel processing of publishing candidates.
           The value '4 ' is specified here as an example.
           The optimal value for this setting depends on your solution and on the CPU capacity of the server that runs the publishing operations.

           For example, if the system architecture leverages a dedicated publishing instance with plenty of CPU capacity, this setting can be
           increased even further.
           On the other hand, on solutions where publishing is run on an authoring server that is already running a high load, changing this
           setting to a non-default value may cause system stability issues. 
           We highly recommend that you perform publishing tests with the expected load before you change this setting in production environments.

           Please remember that your database server's CPU and disk capacity may need to be adjusted to accommodate this setting.
      -->
      <setting name="Publishing.MaxDegreeOfParallelism">
        <patch:attribute name="value">4</patch:attribute>
      </setting>
    </settings>
    <pipelines>
      <publish>
        <!-- This processor performs parallel processing of publishing candidates.
             The Publishing.MaxDegreeOfParallelism setting specifies how many concurrent operations can be run. -->
        <processor type="Sitecore.Publishing.Pipelines.Publish.ProcessQueue, Sitecore.Kernel">
          <patch:attribute name="type">Sitecore.Publishing.Pipelines.ParallelPublish.ParallelProcessQueue,Sitecore.Kernel</patch:attribute>
        </processor>
      </publish>
    </pipelines>
  </sitecore>
</configuration>

From my calculation you are currently getting ~2 items/s which is in line with what I was getting before adding the config. If you can achieve ~100 items/s then you should be able to publish 200,000 items in ~34 minutes.

<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
  <sitecore>
    <settings>
      <!-- This setting is changed from the default value to enable parallel processing of publishing candidates.
           The value '4 ' is specified here as an example.
           The optimal value for this setting depends on your solution and on the CPU capacity of the server that runs the publishing operations.

           For example, if the system architecture leverages a dedicated publishing instance with plenty of CPU capacity, this setting can be
           increased even further.
           On the other hand, on solutions where publishing is run on an authoring server that is already running a high load, changing this
           setting to a non-default value may cause system stability issues. 
           We highly recommend that you perform publishing tests with the expected load before you change this setting in production environments.

           Please remember that your database server's CPU and disk capacity may need to be adjusted to accommodate this setting.
      -->
      <setting name="Publishing.MaxDegreeOfParallelism">
        <patch:attribute name="value">4</patch:attribute>
      </setting>
    </settings>
    <pipelines>
      <publish>
        <!-- This processor performs parallel processing of publishing candidates.
             The Publishing.MaxDegreeOfParallelism setting specifies how many concurrent operations can be run. -->
        <processor type="Sitecore.Publishing.Pipelines.Publish.ProcessQueue, Sitecore.Kernel">
          <patch:attribute name="type">Sitecore.Publishing.Pipelines.ParallelPublish.ParallelProcessQueue,Sitecore.Kernel</patch:attribute>
        </processor>
      </publish>
    </pipelines>
  </sitecore>
</configuration>

From my calculation you are currently getting ~2 items/s which is in line with what I was getting before adding the config. If you can achieve ~100 items/s then you should be able to publish 200,000 items in ~34 minutes.

Edit

I've updated the config and removed a lot of the settings as I did have some issues with it on 9.1. Also the main setting to tweak is the Publishing.MaxDegreeOfParallelism as the others didn't have much effect from my testing.

Source Link
Mark Gibbons
Mark Gibbons
  • 5.7k
  • 1
  • 19
  • 61

I have put together a config from the info on this good answer for Sitecore 8.

Have a try at applying the following config on both CD and CM:

<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
  <sitecore>
    <settings>
      <!-- Enables indexing of cache keys as an experimental optimization to speed up publishing. -->
      <setting name="Caching.CacheKeyIndexingEnabled.AccessResultCache">
        <patch:attribute name="value">true</patch:attribute>
      </setting>
      <!-- Enables indexing of cache keys as an experimental optimization to speed up publishing. -->
      <setting name="Caching.CacheKeyIndexingEnabled.ItemCache">
        <patch:attribute name="value">true</patch:attribute>
      </setting>
      <!-- Enables indexing of cache keys as an experimental optimization to speed up publishing. -->
      <setting name="Caching.CacheKeyIndexingEnabled.ItemPathsCache">
        <patch:attribute name="value">true</patch:attribute>
      </setting>
      <!-- Disables performance counters as an experimental optimization to speed up publishing.
           You should comment out this element if you need to use performance counters. -->
      <setting name="Counters.Enabled">
        <patch:attribute name="value">false</patch:attribute>
      </setting>
      <!-- Disables incremental updates to the link database during publishing operations as an experimental optimization to speed up
           publishing.
           You should comment out this element if you use the LinkDatabase API on your content delivery instances. -->
      <setting name="LinkDatabase.UpdateDuringPublish">
        <patch:attribute name="value">false</patch:attribute>
      </setting>
      <!-- Disables rule item event handlers as an experimental optimization to speed up publishing.
           This setting only affects the instance that runs the publishing operation.
           You should comment out this element if there are rules in any of the following folders that must be executed during publishing:
           /sitecore/system/Settings/Rules/Item Saved/Rules
           /sitecore/system/Settings/Rules/Item Deleted/Rules
           /sitecore/system/Settings/Rules/Version Removed/Rules -->
      <setting name="Rules.ItemEventHandlers.RunDuringPublishing">
        <patch:attribute name="value">false</patch:attribute>
      </setting>
      <!-- This setting is enabled as an experimental optimization to speed up publishing.
           The following configuration limits the execution of rules by item event handlers in the core and master databases.
           You should comment out this element if there are rules in any of the following folders that must be executed on the web database
           or on any custom databases:
           /sitecore/system/Settings/Rules/Item Saved/Rules
           /sitecore/system/Settings/Rules/Item Deleted/Rules
           /sitecore/system/Settings/Rules/Version Removed/Rules -->
      <setting name="Rules.ItemEventHandlers.SupportedDatabases">
        <patch:attribute name="value">core|master</patch:attribute>
      </setting>
      <!-- This setting is changed from the default value to enable parallel processing of publishing candidates.
           The value '4 ' is specified here as an example.
           The optimal value for this setting depends on your solution and on the CPU capacity of the server that runs the publishing operations.

           For example, if the system architecture leverages a dedicated publishing instance with plenty of CPU capacity, this setting can be
           increased even further.
           On the other hand, on solutions where publishing is run on an authoring server that is already running a high load, changing this
           setting to a non-default value may cause system stability issues. 
           We highly recommend that you perform publishing tests with the expected load before you change this setting in production environments.

           Please remember that your database server's CPU and disk capacity may need to be adjusted to accommodate this setting.
      -->
      <setting name="Publishing.MaxDegreeOfParallelism">
        <patch:attribute name="value">4</patch:attribute>
      </setting>
    </settings>
    <pipelines>
      <publish>
        <!-- This processor performs parallel processing of publishing candidates.
             The Publishing.MaxDegreeOfParallelism setting specifies how many concurrent operations can be run. -->
        <processor type="Sitecore.Publishing.Pipelines.Publish.ProcessQueue, Sitecore.Kernel">
          <patch:attribute name="type">Sitecore.Publishing.Pipelines.ParallelPublish.ParallelProcessQueue,Sitecore.Kernel</patch:attribute>
        </processor>
      </publish>
    </pipelines>
  </sitecore>
</configuration>

I have tested this on 9.0.1. I get the following results on a full republish:

  • Local machine - config not applied - ~5 items/s

  • Local machine - config applied - ~1000 items/s

  • Azure PaaS B3, 50 DTU master 50 DTU web - config not applied - ~2 items/s

  • Azure PaaS B3, 50 DTU master 50 DTU web - config applied - ~90 items/s

It hits 100% DTU usage, but CPU usage is not quite maxed so I think the 50 DTU is the bottleneck in this example.

From my calculation you are currently getting ~2 items/s which is in line with what I was getting before adding the config. If you can achieve ~100 items/s then you should be able to publish 200,000 items in ~34 minutes.