9

One of our clients has a large multi-site solution, with 3 sites that have about 100k+ pages each. All three sites have about 3/4 of their contained pages in bucketed items.

To generate the sitemaps for this site, I can think of the following options:

  • Generate each sitemap on publish end for each site that had items published
  • Update sitemap for each item that has been published (add/delete/rename/no change) once the item finished publishing
  • Update each sitemap on manual trigger
  • Update each sitemap via a scheduled task
  • Generate each sitemap at runtime and cache

I don't think the options that happen after publish are super feasible - the site is pretty big and that adds a lot of weight to publish. Plus, coding in the "add/delete/rename/no change" logic for updating a sitemap just seems like way more effort than it should be.

We tried to go the route of generating the sitemap at runtime and caching it, but we started getting sporadic 504 errors on some of the "first loads" because of timeouts while the sitemap was trying to generate.

I am leaning towards the scheduler route. Does anyone have any other implementations that they have done for situations like this that may be better?

Update: please note that I am really looking for code-solutions, rather than modules. However, if you want to link to a module and provide a suggestion based on a highlighted bit of code from the module's implementation that that is more than welcome!

8

I have done this before by using a custom handler to provide the sitemap. In the handler I used the SearchAPI to pull a list of all items that are derived from a _Sitemap template and that have the field in that template (Hide from Sitemap) set to false.

The output of the handler is stored in the Sitecore HtmlCache, so on a publish the cache is cleared and the next time the sitemap is requested it updates.

Here is a really rough example of how it can work:

public class SitemapHandler : IHttpHandler
{
    private readonly Stack<SitemapNode> siteNodes = new Stack<SitemapNode>(); 

    public void ProcessRequest(HttpContext context)
    {
        Assert.ArgumentNotNull(context, "context");
        if (this.DoProcessRequest(context))
        {
            return;
        }

        context.Response.StatusCode = 404;
        context.Response.ContentType = "text/html";
    }

    private bool DoProcessRequest(HttpContext context)
    {
        Assert.ArgumentNotNull((object)context, "context");

        var htmlCache = CacheManager.GetHtmlCache(Sitecore.Context.Site);
        var cacheKey = $"{Sitecore.Context.Site.Name}__sitemap";
        var xml = htmlCache?.GetHtml(cacheKey) ?? string.Empty;


        if (string.IsNullOrWhiteSpace(xml))
        {
            var sitemapNodes = this.GetSitemapNodes(); // this gets all the items from Sitecore that need to go into the sitemap 
            xml = this.GetSitemapDocument(sitemapNodes);
            htmlCache?.SetHtml(cacheKey, xml);
        }

        context.Response.Write(xml);
        context.Response.StatusCode = 200;
        context.Response.ContentType = "text/xml";

        return true;
    }

    public string GetSitemapDocument(IEnumerable<SitemapNode> sitemapNodes)
    {
        XNamespace xmlns = "http://www.sitemaps.org/schemas/sitemap/0.9";

        var root = new XElement(xmlns + "urlset");

        foreach (var sitemapNode in sitemapNodes)
        {
            var urlElement = new XElement(xmlns + "url", new XElement(xmlns + "loc", Uri.EscapeUriString(sitemapNode.Url)), sitemapNode.LastModified == null ? null : new XElement(xmlns + "lastmod", sitemapNode.LastModified.Value.ToLocalTime().ToString("yyyy-MM-ddTHH:mm:sszzz")), sitemapNode.Frequency == null ? null : new XElement(xmlns + "changefreq", sitemapNode.Frequency.Value.ToString().ToLowerInvariant()), sitemapNode.Priority == null ? null : new XElement(xmlns + "priority", sitemapNode.Priority.Value.ToString("F1", CultureInfo.InvariantCulture)));
            root.Add(urlElement);
        }
        var document = new XDocument(root);
        return document.ToString();
    }
}

To get that responding when you hit /sitemap.xml you can just add it to the custom handlers in the sitecore config:

<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
    <sitecore>
        <customHandlers>
            <handler trigger="sitemap.xml" handler="sitemap.ashx" />
        </customHandlers>
    </sitecore>
</configuration>

And then add the handler to the web.config in system.webServer:

<add name="SitemapXml" path="sitemap.ashx" verb="GET" type="MyProject.Handlers.SitemapHandler" preCondition="integratedMode,runtimeVersionv4.0" />

You are still going to get a big hit for a large site on an uncached sitemap tho. This was used for a smaller site that updated the content a lot, there were multiple sitemaps specific for news articles and videos, so it needed to be updated often.

You could run this on a scheduled task or as an async job post publish.

4
  • Great answer, Richard! Is the idea here that the Sitemap would still be generated on request, but that the scheduled task would request and cache it then? – Zachary Kniebel Nov 15 '16 at 19:55
  • Yeah, you could do that, run it on a fire and forget job when a publish finishes. The downside would be in a scaled delivery environment because the cache is not shared, so you would either want to store it in a shared cache or hit all delivery servers in the job. – Richard Seal Nov 15 '16 at 20:02
  • Yeah, that's what I was thinking about too. – Zachary Kniebel Nov 15 '16 at 20:04
  • 1
    I've also seen solutions that store the url in a custom field in the Sitecore index. Then you just have to do a search where that field is not empty to get all the nodes you need. This moves the logic of 'which items do I need to generate a sitemap url for' to a computedindexfield and will make the Sitemap dependant on the index off course. – Koen Heye Nov 16 '16 at 7:52
4

Have you tried the Ultimate Sitemap module?

I'm not sure if you can schedule tasks for it out of the box to be honest. But I think you can get that with the Automated publisher module as well.

I myself have tried the Ultimate Sitemap module and liked it to perform tasks like you described.

2
  • TBH I'm not really looking to use a sitemap module. I don't mind looking at some for implementation ideas, but I am really looking for code more than anything else. Still, this recommendation might be helpful for another so +1 :) – Zachary Kniebel Nov 15 '16 at 17:34
  • sure. you can also reach out to Vasiliy Fomichev who developed the module and get the source code of it. Maybe that helps with ideas... – Diego Nov 15 '16 at 17:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.