2

Using Sitecore 10.1 with Commerce 10.1

There is a field Robots Content in this item - /sitecore/content/Tenant/MySite/Settings.
This is a container setup and creating a file is not an option.

When a user browses mysite/robots.txt, I want the robots.txt to have whatever is entered in the CMS field.

This outcome is not consistent.

If the content is this:

User-agent:*
Crawl-delay: 10
Sitemap: https://example.com

The output is

Sitemap: https://example.com

There is some logic happening internally here and it is removing content. Is there any documentation explaining this. Is there any way to achieve what I need without overwriting the processor.

UPDATE:

I have realized that the content in the robots.txt page seems to be from cache. I had to do an iisreset for every CMS change.

But this issue remains.

When I add Crawl-delay:10 it is not showing in the robots page. That part is being removed.

Improve this question
1
  • I have the same issue as you. I noticed this little puppy: Sitecore.XA.Feature.SiteMetadata.Models.Robots.RobotsRecord. So I think you can only use Allow and Disallow(And Sitemap) Like this: User-agent: GoogleBot Allow: / Sitemap: someUrl
    – gorhal
    Commented May 3, 2022 at 10:47

1 Answer 1

Reset to default
2

Default SXA process for robots.txt is:

  <getRobotsContent>
    <processor type="Sitecore.XA.Feature.SiteMetadata.Pipelines.GetRobotsContent.GetContentFromSettings, Sitecore.XA.Feature.SiteMetadata" resolve="true" />
    <processor type="Sitecore.XA.Feature.SiteMetadata.Pipelines.GetRobotsContent.GetDefaultRobotsContent, Sitecore.XA.Feature.SiteMetadata" resolve="true" />
    <processor type="Sitecore.XA.Feature.SiteMetadata.Pipelines.GetRobotsContent.AppendSitemapUrl, Sitecore.XA.Feature.SiteMetadata" resolve="true" />
  </getRobotsContent>

First it gets content from settings item - make sure that url which you use points to the right site, instead of default website or other site.

Then (only if settings content for robots is empty) it reads content from physical robots.txt file if it exists.

And in the end, if settings of the site defines Sitemap Status as active, it adds empty line plus Sitemap: ... line.

In your case it should render content of the settings field. Why it doesn't? Maybe it's not published? Maybe wrong language version (I don't remember if field is shared)? Maybe it's matching wrong site? Maybe try adding ?sc_site=yoursitename in the url after robots.txt?

Improve this answer
3
  • The issue was, the content was coming from cache and for each CMS change I had to do an iisreset. When I add Crawl-delay:10 it is not showing in the robots page.
    – sukesh
    Commented Apr 8, 2022 at 5:05
  • How can this be an "Answer"?. It does not answer the question. How to add Crawl-delay: 10
    – gorhal
    Commented May 3, 2022 at 10:43
  • Hi @gorhal . The original answer did not include the part about "Crawl-delay: 10". That part was added by OP one day after I answered the original question. My answer describes the mechanism of robots.txt generation in SXA solution - it works for OP if they restarts the app pool. Looks like OP has problems with caching (again, update after my answer was posted). I never investigated caching issues related to robots.txt. Maybe you have more time and can investigate it? And removing cache-delay is new to me as well.
    – Marek Musielak ♦
    Commented May 3, 2022 at 17:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.