Unexpected merge of robots.txt from different sites

Question

In a SXA based multi-site environment, the values of the 'Robots content' (/sitecore/templates/Project/SampleTenant/Settings) fields are unexpectedly merged from two different site.

Let's say the 'Robots content' of site A looks like this

User-agent: *
Disallow: /temp_site_a/

and the 'Robots content' of the site B like this

User-agent: *
Disallow: /temp_site_b/

In this case, the robots.txt for site B would look like this:

User-agent: *
Disallow: /temp_site_b/
Disallow: /temp_site_a/

Sitemap: http://siteb.sc/sitemap.xml

Answer 1

As soon as you configure a SXA site with a wildcard as hostname, the 'Robots content' of this site will be merged into the robots.txt of every site, tenant overlapping.

I'm not sure if it's a feature or a bug, but I definitely didn't expect it and it took a long time to find the cause of the wrong content in our robots.txt.