Why is Yahoo Indexing Bot considered as "evil"?
- by bigstylee
After reading and commenting on this question PHP Library for Keeping your site index by Google, Bing, etc, I was curious to look at StackOverFlow's sitemap. This returned a 404 error which I am guessing is just a protected page by determining if your are a Index Bot or simply doesnt exists.
This then lead me to look at the robots.txt for StackOverFlow. I was surprised to see the comment "Yahoo bot is evil" along with a couple other Indexing bots (Spinn3r and KSCrawler) . I am unfamilular with Spinn3r and KSCrawler but my question is, why are these bots (particular Yahoo) considered as evil? Surely any and all indexing of any Search Engine is a good thing?