on the settings of my site the robots txt does not seem to work for bing and yahoo , but works as expected for google , am i missing anything ??

enter image description here

Robots txt

enter image description here

  • can you please confirm below. 1. google is able to detect? 2. when u submit from yahoo, did u checked the logs what is coming out? Nov 25 '19 at 12:26
  • What happens if change the file contents to "User-agent: * Disallow:" Nov 25 '19 at 12:37
  • Hi @Marcel , thanks for your reply , i want the crawlers to disallow /sitecore , i have not tried this Nov 25 '19 at 13:14
  • 1
    I would suggest that as long as you can confirm that the /robots.txt is outputting what you would expect (e.g. what's in the field shown in the image), then this is not a Sitecore issue.
    – Kasaku
    Nov 25 '19 at 18:27
  • 4
    I'm voting to close this question as off-topic because , this is general issue with a particluar search engines , not a Sitecore Issue
    – Abhay Dhar
    Nov 25 '19 at 19:39

Browse other questions tagged or ask your own question.