How to apply robots.txt file Subdomain Site

Question

We've implemented the application across multiple site, adopting the subdomain model.

In essence, we have the primary site and its associated sub-sites.

The primary site is developed using SXA, while the sub-sites are developed without it. On my domain, https://acd.com, the main site is equipped with a robot.txt file, whereas the sub-site URL is https://sub.acd.com/pdd serving as the homepage.

We've implemented a robot.txt file for the main site.

How can I implement a robot.txt file for the sub-site?

Answer 1

• Create a new Template with a field named RobotsContent.
• Inherit the above template on the Home page template of both Site nodes.
• Create a patch file: This patch file will have one pre-processor to allow the URL extension for the Txt file to serve the Robots txt file.

<pipelines>
    <preprocessRequest>
        <processor type="Sitecore.Pipelines.PreprocessRequest.FilterUrlExtensions, Sitecore.Kernel">
            <param desc="Allowed extensions (comma separated)">aspx, ashx, asmx, txt, xml</param>
        </processor>
    </preprocessRequest>
</pipelines>

• Create a class in your project for the custom processor and use the below code

public class RobotsSitemapTextProcessor : HttpRequestProcessor
    {
        public override void Process(HttpRequestArgs args)
        {
            try
            {
                HttpContext currentContext = HttpContext.Current;
                if (currentContext == null) return;

                string requestedURL = currentContext.Request.Url.ToString().ToLower().Trim();
                if (string.IsNullOrWhiteSpace(requestedURL) || (!requestedURL.EndsWith("robots.txt"))) return;

                var isRobotsRequest = requestedURL.EndsWith("robots.txt");
                var defaultContent = isRobotsRequest ? $"User-agent: * {Environment.NewLine} Disallow: /sitecore" :"";

                if (Sitecore.Context.Site != null && Sitecore.Context.Database != null)
                {
                    Item homeNode = Sitecore.Context.Database.GetItem(Sitecore.Context.Site.StartPath);
                    if (homeNode != null)
                    {
                        var contentField = isRobotsRequest ? homeNode.Fields["RobotsContent"] : string.Empty;

                        if (contentField != null && !string.IsNullOrWhiteSpace(contentField.Value))
                            defaultContent = contentField.Value;
                    }
                }

                currentContext.Response.ContentType = isRobotsRequest ? "text/plain" : "";
                currentContext.Response.Write(defaultContent);
                currentContext.Response.End();
            }
            catch (Exception ex)
            {
                Log.Error("Robots", ex);
            }
        }
    }

• Deploy your application and browse pages: Website-a.com/robots.txt Website-b.com/ robots.txt

See the blog post for more details: Sitemap and Robots in Sitecore Multisite

Answer 2

Since you mentioned one of your Site is hosted on SXA, therefore you will need to create a custom robots handler that will only run for non-SXA sites.

1. Create the custom robots handler as below. Inside this handler there is a check whether the site is a SXA site. Only when the site is non-SXA it will continue the execution.

public class CustomRobotsHandler : HttpRequestProcessor
{
   public int CacheExpiration { set; get; }
   protected IContext Context { get; } = ServiceProviderServiceExtensions.GetService<IContext>(ServiceLocator.ServiceProvider);
   public override void Process(HttpRequestArgs args)
   {
      Uri url = HttpContext.Current.Request.Url;
      if (!url.PathAndQuery.EndsWith("/robots.txt", StringComparison.OrdinalIgnoreCase))
        return;
      SiteInfo siteInfo = this.Context.Site.SiteInfo;
      if (siteInfo == null || this.Context.Site.IsSxaSite())
        return;
      string str1 = string.Format("{0}/{1}/{2}/{3}", (object)"XA-ROBOTS", (object)this.Context.Database?.Name, (object)siteInfo.Name, (object)url);
      object obj = HttpRuntime.Cache[str1];
      string str2;
      if (obj != null)
      {
         str2 = (string)obj;
      }
      else
      {
         str2 = this.GetRobotsContent();
         this.CacheRobotsContent(str1, str2);
      }
      args.HttpContext.Response.ContentType = "text/plain";
      args.HttpContext.Response.Write(str2);
      args.HttpContext.Response.End();
      args.AbortPipeline();
   }
   protected virtual string GetRobotsContent()
   {
      Item item = Sitecore.Context.Database.GetItem(this.Context.Site.StartPath);
      return item.Fields["Robots"].Value;
   }
   protected virtual void CacheRobotsContent(string cacheKey, string robotsContent) => HttpRuntime.Cache.Insert(cacheKey, (object)robotsContent, (CacheDependency)null, DateTime.UtcNow.AddMinutes((double)this.CacheExpiration), Cache.NoSlidingExpiration, CacheItemPriority.NotRemovable, (CacheItemRemovedCallback)null);
}

Note: Robots content in the above handler, is being read from the Site Home item. You can set it to read from other item as well.

2. Thereafter patch the above robots handler using the below configuration. This is patched before the SXA robots handler.

<?xml version="1.0"?>
<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
    <sitecore>
       <pipelines>
          <httpRequestBegin>
            <processor type="SitecoreCustomExtensions.Pipelines.CustomRobotsHandler,SitecoreCustomExtensions" patch:before="*[@type='Sitecore.XA.Foundation.SiteMetadata.Pipelines.HttpRequestBegin.RobotsHandler, Sitecore.XA.Foundation.SiteMetadata']" resolve="true">
              <CacheExpiration>720</CacheExpiration>
            </processor>
          </httpRequestBegin>
        </pipelines>
    </sitecore>
</configuration>

Using above changes, you will be able to handle robots content for the non-SXA sites as well.

Hope this helps!!! Let me know in case you face any issues.

Answer 3

You can refer to the Sitecore documentation on configuring the robots.txt file. This may assist you. https://doc.sitecore.com/xp/en/developers/sxa/latest/sitecore-experience-accelerator/configure-the-robots-txt-file.html