Grapeshot crawler ignoring robots.txt
- by QF_Developer
Has anyone come across a crawler called Grapeshot? They are hammering the same page repeatedly on our website. I believe they are looking for ad related keywords, based on previous content ad campaigns. The odd thing is we never ran any such campaigns on the page they are so interested in. We do have only a few pages running AdSense, is this what has attracted Grapeshot?
I've added the following declaration to my robots.txt, but they don't seem to be honouring it?
User-agent: grapeshot
Disallow: /
Any ideas on how to block this nuisance crawler? I'm starting to think the best way is by setting up IP rules in IIS?