How to exclude a specific page item from search engine crawlers?

Question

I have just started my journey as a Sitecore developer. I wanted to exclude a specific page item from Google indexing/crawling. How can we go about this functionality?

Answer 1

You can add a field in your Sitecore item template such as add a checkbox field for "Do not Allow Search Engine to Index Page", based on this field value you can render a meta tag in your HTML page head section:

<META NAME="robots" CONTENT="noindex">

you can find more information here

https://blog.hubspot.com/marketing/how-to-unindex-pages-from-search-engines

Answer 2

• You can manually create robots.txt file and list pages that you want to exclude for search clawlers:

User-agent: *
Disallow: /page1
Disallow: /page2
...etc

• You can use official Sitecore SXA module where you can setup sitemap and many other features: https://doc.sitecore.com/developers/sxa/17/sitecore-experience-accelerator/en/configure-a-sitemap.html.

• You can find sitemap module that is compatible with your sitecore version in sitecore Market place: https://marketplace.sitecore.net/SearchResults#qr=sitemap

Answer 3

Additional, If you only want to prevent Google web crawlers from indexing a page, then you need to add below tag on your page, in the same way as Vipin define in his answer -

<meta name="googlebot" content="noindex">

Instead of a meta tag, you can also return an X-Robots-Tag header with a value of either noindex or none in your response. below is an example of an HTTP response with an X-Robots-Tag instructing crawlers not to index a page:

HTTP/1.1 200 OK
(…)
X-Robots-Tag: noindex
(…)

reference - https://developers.google.com/search/reference/robots_meta_tag