How is robots.txt content created

Question

This is a multisite implementation with sites under one tenant. There is no robots.txt file in the root.

This is what I have tried:

1. I have cleared the content for Robots content field for all sites. On browsing the URL https://mysite.local/robots.txt, the content rendered is:

User-agent: *
Disallow: /
Disallow: /in/

Sitemap: https://mysite.local/sitemap.xml
Sitemap: https://mysite.local/in/sitemap.xml

Could not understand where the entry for Disallow: /in/ is coming from and why are there only 2 sitemap URLs when there are multiple sites.

2. Added robots.txt file in the root with just this Disallow: /includes/. On browsing the URL https://mysite.local/robots.txt, the content rendered is:

User-agent: *
Disallow: /includes/

Sitemap: https://mysite.local/sitemap.xml
Sitemap: https://mysite.dev.local/in/sitemap.xml

3. In each Site Settings' Robots content field, added the value Disallow: /assets-<sitecode>/. e.g.: Disallow: /assets-shared/, Disallow: /assets-intl/, Disallow: /assets-sg/ etc.

On browsing the URL https://mysite.local/robots.txt, the content rendered is:

User-agent: *
Disallow: /assets-in/
Disallow: /assets-intl/

Sitemap: https://mysite.local/sitemap.xml
Sitemap: https://mysite.local/in/sitemap.xml

The issue is not with publishing items as all sites are configured to master.

I could not find the information I was looking for here. Please help me understand the following:

1. Sitecore will only read either from CMS field or the robots.txt file and not both. Is that correct?
2. If Robots content field is empty, Sitecore adds Disallow: / as default. But this prevents Google bots from indexing and crawling the website.
   Sorry, no idea on SEO things, but disallowing shouldn't be the default. Is that correct?
   Is there any config setting that can prevent this, I mean, Robots content is empty and Disalow: / shouldn't be added.
3. As mentioned above, Disallow: /in/ is being added and only two sitemap URLs added even when none of the CMS fields have those values. The system is reading CMS field only from the International & India Site Settings when there are other sites too.
   So, what other places should I check to troubleshoot.

Using Sitecore 10.2

Answer 1

SXA generates the robots.txt file by running the processor Sitecore.XA.Foundation.SiteMetadata.Pipelines.HttpRequestBegin.RobotsHandler in the <httpRequestBegin> pipeline.

If the requested URL ends with /robots.txt and the context website is an SXA site, this processor will try to generate the robots file content for all SXA sites that match the current domain and absolute path. It will run a special pipeline <getRobotsContent> for each site. This pipeline includes 3 processors:

<getRobotsContent>
      <processor type="Sitecore.XA.Foundation.SiteMetadata.Pipelines.GetRobotsContent.GetContentFromSettings, Sitecore.XA.Foundation.SiteMetadata" resolve="true" />
      <processor type="Sitecore.XA.Foundation.SiteMetadata.Pipelines.GetRobotsContent.GetDefaultRobotsContent, Sitecore.XA.Foundation.SiteMetadata" resolve="true" />
      <processor type="Sitecore.XA.Foundation.SiteMetadata.Pipelines.GetRobotsContent.AppendSitemapUrl, Sitecore.XA.Foundation.SiteMetadata" resolve="true" />
</getRobotsContent>

• GetContentFromSettings retrieves content from the field Robots content of Site Settings
• If there is no content in this field, GetDefaultRobotsContent will try to read a physical robots.txt file. If the file does not exist or it is empty, the following content will be added by default:

"User-agent: *\nDisallow: " + args.Site.VirtualFolder

• Then AppendSitemapUrl processor will add a line with sitemap URL for the site

As a final step, RobotsHandler will take generated content for all sites, reorder lines and remove redundant line breaks.

Now, to answer your questions:

1. Yes, this is correct. Sitecore will either read content from the CMS field or the robots.txt file.
2. I think it makes sense to disallow robot indexing by default, otherwise it would be easy to let bots crawl pages or files that should not be indexed and cause a lot of problems with unwanted URLs appearing in search results.
3. This must be happening because of the way how "matching sites" are detected based on their hostnames, ports and virtual folders. It looks like for your request https://mysite.local/robots.txt it finds only two matching sites. Here is the method that is responsible for this:

    protected virtual IList<SiteContext> GetMatchingSites(HttpRequestArgs args)
    {
      Uri uri = new Uri(HttpContext.Current.Request.Url, args.HttpContext.Request.RawUrl);
      return (IList<SiteContext>) this.GetSiteContexts(uri.Host, uri.AbsolutePath, uri.Port).Where<SiteContext>((Func<SiteContext, bool>) (context => context != null)).Where<SiteContext>((Func<SiteContext, bool>) (context => context.IsSxaSite())).ToList<SiteContext>();
    }

    protected virtual IEnumerable<SiteContext> GetSiteContexts(string hostName, string fullPath, int portNumber)
    {
      fullPath = fullPath.ToLowerInvariant();
      foreach (SiteInfo site in this.GetSites())
      {
        fullPath = site.VirtualFolder + fullPath;
        if (site.Matches(hostName, fullPath, portNumber))
          yield return new SiteContext(site);
      }
    }

Please double check your site configuration and compare their hostnames, ports and virtual folders, this can explain why some of the sites are excluded from the RobotsHandler processor.