Options for sitemap generation on larger solutions

Question

One of our clients has a large multi-site solution, with 3 sites that have about 100k+ pages each. All three sites have about 3/4 of their contained pages in bucketed items.

To generate the sitemaps for this site, I can think of the following options:

• Generate each sitemap on publish end for each site that had items published
• Update sitemap for each item that has been published (add/delete/rename/no change) once the item finished publishing
• Update each sitemap on manual trigger
• Update each sitemap via a scheduled task
• Generate each sitemap at runtime and cache

I don't think the options that happen after publish are super feasible - the site is pretty big and that adds a lot of weight to publish. Plus, coding in the "add/delete/rename/no change" logic for updating a sitemap just seems like way more effort than it should be.

We tried to go the route of generating the sitemap at runtime and caching it, but we started getting sporadic 504 errors on some of the "first loads" because of timeouts while the sitemap was trying to generate.

I am leaning towards the scheduler route. Does anyone have any other implementations that they have done for situations like this that may be better?

Update: please note that I am really looking for code-solutions, rather than modules. However, if you want to link to a module and provide a suggestion based on a highlighted bit of code from the module's implementation that that is more than welcome!

Answer 1

I have done this before by using a custom handler to provide the sitemap. In the handler I used the SearchAPI to pull a list of all items that are derived from a _Sitemap template and that have the field in that template (Hide from Sitemap) set to false.

The output of the handler is stored in the Sitecore HtmlCache, so on a publish the cache is cleared and the next time the sitemap is requested it updates.

Here is a really rough example of how it can work:

public class SitemapHandler : IHttpHandler
{
    private readonly Stack<SitemapNode> siteNodes = new Stack<SitemapNode>(); 

    public void ProcessRequest(HttpContext context)
    {
        Assert.ArgumentNotNull(context, "context");
        if (this.DoProcessRequest(context))
        {
            return;
        }

        context.Response.StatusCode = 404;
        context.Response.ContentType = "text/html";
    }

    private bool DoProcessRequest(HttpContext context)
    {
        Assert.ArgumentNotNull((object)context, "context");

        var htmlCache = CacheManager.GetHtmlCache(Sitecore.Context.Site);
        var cacheKey = $"{Sitecore.Context.Site.Name}__sitemap";
        var xml = htmlCache?.GetHtml(cacheKey) ?? string.Empty;


        if (string.IsNullOrWhiteSpace(xml))
        {
            var sitemapNodes = this.GetSitemapNodes(); // this gets all the items from Sitecore that need to go into the sitemap 
            xml = this.GetSitemapDocument(sitemapNodes);
            htmlCache?.SetHtml(cacheKey, xml);
        }

        context.Response.Write(xml);
        context.Response.StatusCode = 200;
        context.Response.ContentType = "text/xml";

        return true;
    }

    public string GetSitemapDocument(IEnumerable<SitemapNode> sitemapNodes)
    {
        XNamespace xmlns = "http://www.sitemaps.org/schemas/sitemap/0.9";

        var root = new XElement(xmlns + "urlset");

        foreach (var sitemapNode in sitemapNodes)
        {
            var urlElement = new XElement(xmlns + "url", new XElement(xmlns + "loc", Uri.EscapeUriString(sitemapNode.Url)), sitemapNode.LastModified == null ? null : new XElement(xmlns + "lastmod", sitemapNode.LastModified.Value.ToLocalTime().ToString("yyyy-MM-ddTHH:mm:sszzz")), sitemapNode.Frequency == null ? null : new XElement(xmlns + "changefreq", sitemapNode.Frequency.Value.ToString().ToLowerInvariant()), sitemapNode.Priority == null ? null : new XElement(xmlns + "priority", sitemapNode.Priority.Value.ToString("F1", CultureInfo.InvariantCulture)));
            root.Add(urlElement);
        }
        var document = new XDocument(root);
        return document.ToString();
    }
}

To get that responding when you hit /sitemap.xml you can just add it to the custom handlers in the sitecore config:

<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
    <sitecore>
        <customHandlers>
            <handler trigger="sitemap.xml" handler="sitemap.ashx" />
        </customHandlers>
    </sitecore>
</configuration>

And then add the handler to the web.config in system.webServer:

<add name="SitemapXml" path="sitemap.ashx" verb="GET" type="MyProject.Handlers.SitemapHandler" preCondition="integratedMode,runtimeVersionv4.0" />

You are still going to get a big hit for a large site on an uncached sitemap tho. This was used for a smaller site that updated the content a lot, there were multiple sitemaps specific for news articles and videos, so it needed to be updated often.

You could run this on a scheduled task or as an async job post publish.

Answer 2

Have you tried the Ultimate Sitemap module?

I'm not sure if you can schedule tasks for it out of the box to be honest. But I think you can get that with the Automated publisher module as well.

I myself have tried the Ultimate Sitemap module and liked it to perform tasks like you described.