0

I have just started my journey as a Sitecore developer. I wanted to exclude a specific page item from Google indexing/crawling. How can we go about this functionality?

2

You can add a field in your Sitecore item template such as add a checkbox field for "Do not Allow Search Engine to Index Page", based on this field value you can render a meta tag in your HTML page head section:

<META NAME="robots" CONTENT="noindex">

you can find more information here

https://blog.hubspot.com/marketing/how-to-unindex-pages-from-search-engines

2
  • You can manually create robots.txt file and list pages that you want to exclude for search clawlers:
User-agent: *
Disallow: /page1
Disallow: /page2
...etc
2

Additional, If you only want to prevent Google web crawlers from indexing a page, then you need to add below tag on your page, in the same way as Vipin define in his answer -

<meta name="googlebot" content="noindex">

Instead of a meta tag, you can also return an X-Robots-Tag header with a value of either noindex or none in your response. below is an example of an HTTP response with an X-Robots-Tag instructing crawlers not to index a page:

HTTP/1.1 200 OK
(…)
X-Robots-Tag: noindex
(…)

reference - https://developers.google.com/search/reference/robots_meta_tag

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.