1

I have a Sitecore 10.2 site that, when accessing the site's robots.txt page, gives me a 404.

I have filled out the Robots section in the Content Editor -> Settings page and have properly saved/published my changes.

Reading about how the robots.txt file is built suggests that something is not set properly, but I'm not sure what. Does anyone have any insight?

3
  • See if there is any additional information/Error logged in the log file. Also, try after recycling the app pool. Commented Jul 10, 2023 at 15:41
  • Do you have any custom processors for 404 Not Found page in the <httpRequestBegin> pipeline? Sometimes they can trigger 404 response if specific URLs such as robots.txt are not excluded from them.
    – Anna Gevel
    Commented Jul 11, 2023 at 7:59
  • Also please check allowed extensions if txt extension is available or not? Commented Jul 11, 2023 at 13:54

1 Answer 1

0

There are a few things to check when troubleshooting 404 errors for robots other than the official one at Configure the robots.txt file.

  1. Please make sure your are accessing the robots file correctly, like https://yourdomain.com/robots.txt (with no language/site prefix).
  2. Check that the site you are accessing is configured as an SXA site in Sitecore. The robots.txt generation only happens automatically for SXA sites.
  3. Verify the settings under Site Settings > Site Metadata > Robots have content and are published. This field provides the content for robots.txt.
  4. Enable debug logging for the Sitecore.XA.Foundation.SiteMetadata namespace and inspect the logs when requesting robots.txt to see exactly what content is being generated.

Also please refer to this one to confirm if you are with same scenario at Unexpected merge of robots.txt from different sites

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.