I have just started my journey as a Sitecore developer. I wanted to exclude a specific page item from Google indexing/crawling. How can we go about this functionality?
You can add a field in your Sitecore item template such as add a checkbox field for "Do not Allow Search Engine to Index Page", based on this field value you can render a meta tag in your HTML page head section:
<META NAME="robots" CONTENT="noindex">
you can find more information here
- You can manually create robots.txt file and list pages that you want to exclude for search clawlers:
User-agent: * Disallow: /page1 Disallow: /page2 ...etc
You can use official Sitecore SXA module where you can setup sitemap and many other features: https://doc.sitecore.com/developers/sxa/17/sitecore-experience-accelerator/en/configure-a-sitemap.html.
You can find sitemap module that is compatible with your sitecore version in sitecore Market place: https://marketplace.sitecore.net/SearchResults#qr=sitemap
Additional, If you only want to prevent Google web crawlers from indexing a page, then you need to add below tag on your page, in the same way as Vipin define in his answer -
<meta name="googlebot" content="noindex">
Instead of a meta tag, you can also return an X-Robots-Tag header with a value of either
noindex or none in your response. below is an example of an HTTP response with an X-Robots-Tag instructing crawlers not to index a page:
HTTP/1.1 200 OK (…) X-Robots-Tag: noindex (…)