Grapeshot crawler ignoring robots.txt
Posted
by
QF_Developer
on Pro Webmasters
See other posts from Pro Webmasters
or by QF_Developer
Published on 2013-10-21T09:33:38Z
Indexed on
2013/10/21
10:16 UTC
Read the original article
Hit count: 310
Has anyone come across a crawler called Grapeshot? They are hammering the same page repeatedly on our website. I believe they are looking for ad related keywords, based on previous content ad campaigns. The odd thing is we never ran any such campaigns on the page they are so interested in. We do have only a few pages running AdSense, is this what has attracted Grapeshot?
I've added the following declaration to my robots.txt, but they don't seem to be honouring it?
User-agent: grapeshot
Disallow: /
Any ideas on how to block this nuisance crawler? I'm starting to think the best way is by setting up IP rules in IIS?
© Pro Webmasters or respective owner