Robots.txt and "Bad" Robots
- by Lynda
I understand robots.txt and its purpose. I have read some people saying that using a Robots.txt gives "bad" robots or robots who do not obey a robots.txt a way to access pages on your site that you do not want accessed. While I am not looking to get into a debate about that I do have a question:
If I have a structure like this:
/Folder/
…