Robots.txt file for multi site sitecore implementation

Question

Is there an example available for robots.txt file on how it should look for a multisite implementation of sitecore?

Since there is a single robots.txt file available on the file system - is there a way of having the required information on a physical file as opposed to generating it on the fly based on the website domain it was requested through?

Answer 1

Sitecore SXA has a good implementation of this. They register it as a HttpRequestBegin pipeline handler.

Very simplified version of how it works (as I understand it)

• Check the request url ends with /robots.txt
• Get the context site and make sure we want to process the request for the given site
• Check the cache for data for the context site, write it to the response if it exists
• Get the content from the context site: /sitecore/content/Tenant/Site/Settings item "Robots content" field
• If it's empty prefill it with a default value (such as deny all)
• Check if a sitemap is available and append it to the content
• Add the content to the cache, and write it to the response

This is a great way as it is high performance and also allows the robots.txt content to be easily editable.

Answer 2

While it is possible to implement a solution with physical files, this approach has some disadvantages:

• To apply a change you have to physically change a file on the server.
• If you have scaled setup with multiple CD server, you would have to apply changes to robots.txt on each of the servers.

In contrary, if you generate robots.txt files on the fly based on a hostname and store robots.txt content in Sitecore items, you would have a central place to manage, which would work across all your CD servers.

If you still wanna go for a file-based solution, you could consider using IIS URL rewrite module:

• Have a robots.txt file per website, e.g site1.robots.txt, site2.robots.txt
• Setup IIS rewrite rules to process the /robots.txt requests and rewrite file path based on the hostname.

From top of my head, this could look something like this:

<rule name="Multisite robots.txt">
  <match url="^robots\.txt$" />
  <action type="Rewrite" url="{HTTP_HOST}.robots.txt" />
</rule>

And if you decide to go for dynamic robots.txt generation, there are tons of examples you can find in the internet, e.g https://sitecoreclimber.wordpress.com/2014/07/27/sitecore-multisite-robots-txt/ or even Marketplace modules https://marketplace.sitecore.net/Modules/U/Ultimate_Sitemap_XML.aspx