3

Is there an example available for robots.txt file on how it should look for a multisite implementation of sitecore?

Since there is a single robots.txt file available on the file system - is there a way of having the required information on a physical file as opposed to generating it on the fly based on the website domain it was requested through?

  • Please specify your question more in detail. This is too broad to answer without lot of assumptions – Peter Procházka Feb 18 '19 at 10:19
  • In this instance, I think you'll have better luck using moz.com/learn/seo/robots-meta-directives meta directives. Can setup a metahead rendering that has a data source that takes the values required to output what you need the directives to do. – Anicho Feb 18 '19 at 10:53
  • 1
    Durga, please do follow Peter's instruction and add more information. This does have potential to be a good question! – Matthew FitzGerald-Chamberlain Feb 18 '19 at 16:31
3

Sitecore SXA has a good implementation of this. They register it as a HttpRequestBegin pipeline handler.

Very simplified version of how it works (as I understand it)

  • Check the request url ends with /robots.txt
  • Get the context site and make sure we want to process the request for the given site
  • Check the cache for data for the context site, write it to the response if it exists
  • Get the content from the context site: /sitecore/content/Tenant/Site/Settings item "Robots content" field
  • If it's empty prefill it with a default value (such as deny all)
  • Check if a sitemap is available and append it to the content
  • Add the content to the cache, and write it to the response

This is a great way as it is high performance and also allows the robots.txt content to be easily editable.

|improve this answer|||||
1

While it is possible to implement a solution with physical files, this approach has some disadvantages:

  • To apply a change you have to physically change a file on the server.
  • If you have scaled setup with multiple CD server, you would have to apply changes to robots.txt on each of the servers.

In contrary, if you generate robots.txt files on the fly based on a hostname and store robots.txt content in Sitecore items, you would have a central place to manage, which would work across all your CD servers.

If you still wanna go for a file-based solution, you could consider using IIS URL rewrite module:

  • Have a robots.txt file per website, e.g site1.robots.txt, site2.robots.txt
  • Setup IIS rewrite rules to process the /robots.txt requests and rewrite file path based on the hostname.

From top of my head, this could look something like this:

<rule name="Multisite robots.txt">
  <match url="^robots\.txt$" />
  <action type="Rewrite" url="{HTTP_HOST}.robots.txt" />
</rule>

And if you decide to go for dynamic robots.txt generation, there are tons of examples you can find in the internet, e.g https://sitecoreclimber.wordpress.com/2014/07/27/sitecore-multisite-robots-txt/ or even Marketplace modules https://marketplace.sitecore.net/Modules/U/Ultimate_Sitemap_XML.aspx

|improve this answer|||||

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.