Publishing items in bulk

Question

I need to publish around 100K items in Sitecore 8.1 programmatically. They don't have a common parent, they are spread out in several folders but I have their IDs.

I tried using PublishOptions:

Sitecore.Publishing.PublishOptions publishOptions = new Sitecore.Publishing.PublishOptions(item.Database,                                           
                                     Sitecore.Data.Database.GetDatabase("web"),
                                     Sitecore.Publishing.PublishMode.SingleItem,
                                     item.Language,
                                     System.DateTime.Now);
Sitecore.Publishing.Publisher publisher = new Sitecore.Publishing.Publisher(publishOptions);
publisher.Options.RootItem = item;
publisher.Options.Deep = false;
publisher.Publish();

and PublishItem:

PublishManager.PublishItem(item, new Data.Database[] { Sitecore.Data.Database.GetDatabase("web") }, item.Languages, false, false, true);

Of course, each of the above code snippets is surrounded by a foreach.

I'm not sure which is better in theory, but I see that both of them:

1. Are slow
2. Create thousands of new publish and indexing jobs
3. After publishing 20k-30k all jobs disappear as if Sitecore server resets

So am I doing something wrong or is there a better way of doing this?

Answer 1

Publishing that many items will always be quite slow, there is a lot of reading, writing and indexing that is required however. I don't think there is going to be any code which will solve this.

I've not seem all the jobs disappearing (with the exception of, as you say Sitecore restarting - can you see this happening in the logs?). However there are some Sitecore features that might be able to help outside the scope of code changes:

8.2 Publishing Service module

As I'm sure you already are aware 8.2 has the new Publishing Service module which should stop this being an issue:

"The module answers inherent problems with Publishing large amounts of data (100,000’s of items) across multiple data-centers and geo-graphic locations"

8.1 also has a neat package which (at least in theory) makes the upgrade to 8.2 much more simplistic.

Solutions in 8.1

That said it is still an upgrade and so it will undoubtedly be painful! From 7.2 onwards you can make it significantly quicker as long as you have some horsepower in your CM server.

Goooo Parallel

Rename the following file

Sitecore.Publishing.Parallel.config.disabled

to

Sitecore.Publishing.Parallel.config

This enables you to change parallel publishing options and takes advantages of all the cores of your server.

<!--  PUBLISHING MAX DEGREE OF PARALLELISM
  If set to a positive number, it limits the number of concurrent publishing operations to the set value.
  If it is -1, there is no limit on the number of concurrently running operations.
  Default value: 1
-->
<setting name="Publishing.MaxDegreeOfParallelism" value="1" />

Publishing Optimisations

Rename:

Sitecore.Publishing.Optimizations.config.example

to

Sitecore.Publishing.Optimizations.config

In here there are a bunch of optimisations you can tweak (and possibly break your site - so go careful), full details are in the references below.

One example is:

Disable Link Database

As mentioned in the article below you can also disable the link database being updated during the publish:

LinkDatabase.UpdateDuringPublish=false

Fast servers...

All these optimisations are reliant on having beefy CM boxes, if they are not full of fast disks and lots of cores then you might not see great improvements.

References

• Enabling Parallel Publishing with Sitecore 7.2

• Dedicated Publishing Instance Configuration

Answer 2

Don't know if it is feasible for you, but maybe you want to consider upgrading to 8.2 so you can use the new Publishing Service. This is much faster as the current publish and should be able to handle this.

More info here: https://dev.sitecore.net/Downloads/Sitecore_Publishing_Service/11/Sitecore_Publishing_Service_11_Initial_Release.aspx

Answer 3

Another possible solution is to build the Publish Queue programatically, then publish all at once as a single Publish operation. This method will then execute the publish:end and publish:end:remote events once. Events such as clearing the cache, updating indexes, then only happen once. It should be very quick.

I created some utilities to assist a while back: https://gist.github.com/jraps20/638312ea825f9a5ca5ee8da2c6ff92c1

public static class PublishUtils
{
    public static void CreateAndPublishQueue(Database sourceDatabase, Database[] targetDatabases, Language[] targetLanguages, IEnumerable<ID> itemIds, bool skipEvents = false, bool useSecurityDisabler = true)
    {
        Assert.IsNotNull(sourceDatabase, "sourceDatabase");
        Assert.IsNotNull(targetDatabases, "targetDatabases");
        Assert.IsNotNull(targetLanguages, "targetLanguages");

        var publishingCandidates = SetPublishingCandidates(sourceDatabase, targetDatabases, targetLanguages, itemIds);

        ProcessCandidates(sourceDatabase, targetDatabases, targetLanguages, publishingCandidates, skipEvents, useSecurityDisabler);
    }

    private static List<PublishingCandidate> SetPublishingCandidates(Database sourceDatabase, Database[] targetDatabases, Language[] targetLanguages, IEnumerable<ID> itemIds)
    {
        var publishingCandidates = new List<PublishingCandidate>();

        foreach (var itemId in itemIds)
        {
            publishingCandidates.AddRange(CreatePublishingCandidatesFromItemId(sourceDatabase, targetDatabases, targetLanguages, itemId));
        }

        return publishingCandidates;
    }

    public static List<PublishingCandidate> CreatePublishingCandidatesFromItemId(Database sourceDatabase, Database[] targetDatabases, Language[] targetLanguages, ID itemId)
    {
        if (sourceDatabase == null)
        {
            Log.Info("sourceDatabase == null", new object());
            return Enumerable.Empty<PublishingCandidate>().ToList();
        }
        if (targetDatabases == null)
        {
            Log.Info("targetDatabases == null", new object());
            return Enumerable.Empty<PublishingCandidate>().ToList();
        }
        if (targetLanguages == null)
        {
            Log.Info("targetLanguages == null", new object());
            return Enumerable.Empty<PublishingCandidate>().ToList();
        }

        var publishingCandidates = new List<PublishingCandidate>();

        var item = sourceDatabase.GetItem(itemId);

        if (item == null)
        {
            Log.Info("item == null", new object());
            return Enumerable.Empty<PublishingCandidate>().ToList();
        }

        foreach (var publishingTargetDatabase in targetDatabases.Where(tDb => tDb != null))
        {
            foreach (var publishingTargetLanguage in targetLanguages.Where(lang => lang != null))
            {
                var publishOptions = new PublishOptions(
                    Database.GetDatabase("master"),
                    publishingTargetDatabase,
                    PublishMode.Full,
                    publishingTargetLanguage,
                    DateTime.Now)
                {
                    CompareRevisions = false
                };

                publishingCandidates.Add(new PublishingCandidate(itemId, publishOptions));
                Log.Info($"Added item {item.Paths.FullPath} to the publish queue, target language: {publishingTargetLanguage}, targetDatabase: {publishingTargetDatabase}", new object());
            }
        }

        return publishingCandidates;
    }

    public static void ProcessCandidates(Database sourceDatabase, Database[] targetDatabases, Language[] targetLanguages, List<PublishingCandidate> publishingCandidates, bool skipEvents = false, bool useSecurityDisabler = true)
    {
        Assert.IsNotNull(sourceDatabase, "sourceDatabase");
        Assert.IsNotNull(targetDatabases, "targetDatabases");
        Assert.IsNotNull(targetLanguages, "targetLanguages");

        Log.Info($"{publishingCandidates.Count} items added to publish queue", new object());

        var triggerDatabase = targetDatabases.First();
        var triggerLanguage = targetLanguages.First();

        var defaultPublishOptions = new PublishOptions(
                sourceDatabase,
                triggerDatabase, // need to pass database to satisfy contructor
                PublishMode.Full,
                triggerLanguage, // need to pass language to satisfy contructor
                DateTime.Now)
             {
                CompareRevisions = false
             };

        var publishContext = PublishManager.CreatePublishContext(defaultPublishOptions);

        // required or null exception thrown
        publishContext.Languages = targetLanguages;

        publishContext.Queue.Add(publishingCandidates);

        if (skipEvents)
        {
            var queue = new ProcessQueue();
            Log.Info("Processing Publish Queue, skipEvents = true", new object());

            if (useSecurityDisabler)
            {
                Log.Info("Processing Publish Queue, security disabled", new object());
                using (new SecurityDisabler())
                    queue.Process(publishContext);
            }
            else
            {
                Log.Info("Processing Publish Queue, security enabled", new object());
                queue.Process(publishContext);
            }
        }
        else
        {
            Log.Info("Processing Publish Queue, skipEvents = false", new object());

            if (useSecurityDisabler)
            {
                Log.Info("Processing Publish Queue, security disabled", new object());
                using (new SecurityDisabler())
                    CorePipeline.Run("publish", publishContext);
            }
            else
            {
                Log.Info("Processing Publish Queue, security enabled", new object());
                CorePipeline.Run("publish", publishContext);
            }
            Log.Info("Raising publish:end event", new object());
            global::Sitecore.Events.Event.RaiseEvent("publish:end", new Publisher(defaultPublishOptions));
            Log.Info("Raising publish:end:remote event", new object());
            global::Sitecore.Events.Event.RaiseEvent("publish:end:remote", new Publisher(defaultPublishOptions));
        }
        Log.Info("Finished processing Publish Queue", new object());
    }
}

Usage

// set target database(s)
Database[] targetDatabases = { Database.GetDatabase("web")};

// get languages
var source = new List<Language>();
foreach (var database in targetDatabases)
{
  source.AddRange(LanguageManager.GetLanguages(database));
}
var languages = source.Distinct().ToArray();

// set source database
var sourceDatabase = Database.GetDatabase("master");

// set Item Ids required to be published
ID[] ids = <populate with item ids>

// publish all as one operation
PublishUtils.CreateAndPublishQueue(sourceDatabase, targetDatabases, languages, ids);