Collection was modified; enumeration operation may not execute

Question

We are installing Sitecore 9.3 SXA Vanilla instance and after completing the installation I have created an aspx page that will be executed from PowerShell and that will install some packages on the master DB and publish all the items in the package from master to Web Database. But after 2-3 package install, it fails with the error mentioned below in the logs, also, the packages which got installed are not completely published to the Web Database. I can only see a few items.

Code used for package install and publishing

using (new SecurityDisabler())
{
    using (new SyncOperationContext())
    {
        IProcessingContext context = new SimpleProcessingContext();
        IItemInstallerEvents itemInstallerEvents = new DefaultItemInstallerEvents(new BehaviourOptions(InstallMode.Merge, MergeMode.Merge));
        context.AddAspect(itemInstallerEvents);

        IFileInstallerEvents fileInstallerEvents = new DefaultFileInstallerEvents(true);
        context.AddAspect(fileInstallerEvents);
        new Sitecore.Install.Installer().InstallPackage(MainUtil.MapPath(packagePath), context);
    }

    if (publish)
    {

        List<Item> items = GetItemsFromPackage(packagePath);

        var targets = ConfigurationManager.AppSettings[Constants.CONFIG_SITECORE_PUBLISHING_TARGETS].ToString().Split('|').ToList();

        foreach (string target in targets)
        {
            var targetDatabase = Sitecore.Configuration.Factory.GetDatabase(target);

            installLog += "INSTALL PUBLISH to: " + targetDatabase.Name + "<br>";

            // Publish items
            foreach (var item in items)
            {
                PublishOptions publishOptions = new PublishOptions(item.Database, targetDatabase, PublishMode.SingleItem, item.Language, DateTime.Now);
                publishOptions.UserName = "sitecore\\admin";
                publishOptions.RootItem = item;
                publishOptions.Deep = true;
                Publisher publisher = new Publisher(publishOptions);
                installLog += "INSTALL PUBLISHING: " + item.Paths.FullPath + "<br>";
                publisher.PublishAsync();
                item.Publishing.ClearPublishingCache();
            }
        }
    }
}

Logs

An exception has occurred.
    ExceptionType: 'InvalidOperationException'
    ExceptionMessage: 'Collection was modified; enumeration operation may not execute.'
    StackTrace: '   at System.Collections.Hashtable.HashtableEnumerator.MoveNext()
           at Sitecore.Collections.SafeDictionary`2.<GetEnumerator>d__29.MoveNext()
           at Sitecore.Collections.StringDictionary.ToNameValueCollection()
           at Sitecore.Web.SiteInfo..ctor(StringDictionary properties)
           at Sitecore.Sites.DefaultSiteContextFactory.GetSites()
           at Sitecore.Sites.DefaultSiteContextFactory.GetSiteNames()
           at Sitecore.Configuration.DefaultFactory.GetSiteNames()
           at Sitecore.Configuration.DefaultFactory.GetSiteInfoList()
           at Sitecore.Caching.ItemCache.ClearFilterCaches()
           at System.EventHandler`1.Invoke(Object sender, TEventArgs e)
           at Sitecore.Data.Engines.EngineCommand`2.RaiseEvent[TArgs](EventHandler`1 handlers, Func`2 argsCreator)
           at Sitecore.Data.Engines.EngineCommand`2.Execute()
           at Sitecore.Data.Engines.DataEngine.SaveItem(Item item)
           at Sitecore.Pipelines.ItemProvider.SaveItem.TriggerDataEngine.Process(SaveItemArgs args)
           at (Object , Object )
           at Sitecore.Pipelines.CorePipeline.Run(PipelineArgs args)
           at Sitecore.Data.Managers.DefaultItemManager.SaveItem(Item item)
           at Sitecore.Data.Items.ItemEditing.AcceptChanges(Boolean updateStatistics, Boolean silent)
           at Sitecore.Data.Items.ItemEditing.EndEdit(Boolean updateStatistics, Boolean silent)
           at Sitecore.Install.Items.ItemInstaller.VersionInstaller.InstallVersion(Item version, IProcessingContext context)
           at Sitecore.Install.Items.ItemInstaller.VersionInstaller.PasteVersion(XmlNode versionXml, Item target, VersionInstallMode mode, IProcessingContext context, Boolean removeOtherVersions)
           at Sitecore.Install.Items.ItemInstaller.InstallEntry(PackageEntry entry)
           at Sitecore.Install.Items.ItemInstaller.Flush()
           at Sitecore.Install.Framework.SinkDispatcher.Flush()
           at Sitecore.Install.Utils.EntrySorter.Flush()
           at Sitecore.Install.Zip.PackageReader.Populate(ISink`1 sink)
           at Sitecore.Install.Utils.EntrySorter.Populate(ISink`1 sink)
           at Sitecore.Install.Installer.InstallPackage(String path, Boolean registerInstallation, ISource`1 source, IProcessingContext context)
           at Sitecore.Install.Installer.InstallPackage(String path, IProcessingContext context)

Any thoughts!! Thanks.

Answer 1

As mentioned in the comment, instead of triggering a publish for each item, you may trigger it via an incremental publish.

You need to remove the ForLoop on the items. Below is an updated code snippet

var languageList = new List<Language>();

Database db = Factory.GetDatabase(this.sourceDatabase);
foreach (Language language in LanguageManager.GetLanguages(db))
{
    languageList.Add(language);
}

var sourceDatabase = Sitecore.Configuration.Factory.GetDatabase("master");

foreach (string target in targets)
{
    var targetDatabase = Sitecore.Configuration.Factory.GetDatabase(target);

    PublishOptions publishOptions = new PublishOptions(sourceDatabase, targetDatabase, PublishMode.Incremental, languageList.FirstOrDefault(), DateTime.Now);
    publishOptions.UserName = "sitecore\\admin";
    
    Publisher publisher = new Publisher(publishOptions, languageList);
    
    publisher.PublishAsync();
}

It will perform an incremental publish on all the languages and each installed item will be published.

The advantage here is that your cache will be cleared once instead after each item published.

Note I have not yet tested the code, but this is the main principle how to trigger the incremental publish for each target database that you have.