Grapeshot crawler ignoring robots.txt

Posted by QF_Developer on Pro Webmasters See other posts from Pro Webmasters or by QF_Developer
Published on 2013-10-21T09:33:38Z Indexed on 2013/10/21 10:16 UTC
Read the original article Hit count: 310

Filed under:
|
|

Has anyone come across a crawler called Grapeshot? They are hammering the same page repeatedly on our website. I believe they are looking for ad related keywords, based on previous content ad campaigns. The odd thing is we never ran any such campaigns on the page they are so interested in. We do have only a few pages running AdSense, is this what has attracted Grapeshot?

I've added the following declaration to my robots.txt, but they don't seem to be honouring it?

User-agent: grapeshot
Disallow: /

Any ideas on how to block this nuisance crawler? I'm starting to think the best way is by setting up IP rules in IIS?

© Pro Webmasters or respective owner

Related posts about robots.txt

Related posts about web-crawlers