In a SXA based multi-site environment, the values of the 'Robots content' (/sitecore/templates/Project/SampleTenant/Settings) fields are unexpectedly merged from two different site.
Let's say the 'Robots content' of site A looks like this
User-agent: * Disallow: /temp_site_a/
and the 'Robots content' of the site B like this
User-agent: * Disallow: /temp_site_b/
In this case, the robots.txt for site B would look like this:
User-agent: * Disallow: /temp_site_b/ Disallow: /temp_site_a/ Sitemap: http://siteb.sc/sitemap.xml