Is there any reason to allow Yahoo! Slurp to crawl my site?

Posted by James Skemp on Pro Webmasters See other posts from Pro Webmasters or by James Skemp
Published on 2011-11-27T00:09:56Z Indexed on 2011/11/27 2:02 UTC
Read the original article Hit count: 404

Filed under:
|

I thought a year or more ago Yahoo! would be using another search engine for results, and no longer using their own Slurp bot. However, a couple of the sites I manage Yahoo! Slurp continues to crawl pages, and seems to ignore the Gone status code when returned (as it keeps coming back).

Is there any reason why I wouldn't want to block Yahoo! Slurp via robots.txt or by IP (since it tends to ignore robots.txt in some cases anyways)?

I've confirmed that when the bot does hit it is from Yahoo! IPs, so I believe this is a legit instance of the bot.

Is Yahoo Search the same as Bing Search now? is a related question, but I don't think it completely answers whether one should add a new block of the bot.

© Pro Webmasters or respective owner

Related posts about yahoo

Related posts about spider