Grapeshot crawler ignoring robots.txt

Posted by QF_Developer on Pro Webmasters See other posts from Pro Webmasters or by QF_Developer
Published on 2013-10-21T09:33:38Z Indexed on 2013/10/21 10:16 UTC
Read the original article Hit count: 312

Filed under:

robots.txt

|

web-crawlers

|

spam

Has anyone come across a crawler called Grapeshot? They are hammering the same page repeatedly on our website. I believe they are looking for ad related keywords, based on previous content ad campaigns. The odd thing is we never ran any such campaigns on the page they are so interested in. We do have only a few pages running AdSense, is this what has attracted Grapeshot?

I've added the following declaration to my robots.txt, but they don't seem to be honouring it?

User-agent: grapeshot
Disallow: /

Any ideas on how to block this nuisance crawler? I'm starting to think the best way is by setting up IP rules in IIS?

© Pro Webmasters or respective owner

Related posts about robots.txt

Robots.txt practices with .htaccess redirections (inherits)

as seen on Pro Webmasters - Search for 'Pro Webmasters'
I have a question regarding how to write robots.txt files for many domains and subdomains with redirects in place. We have a hosting account that enacts primary and add-on domains. All of our domains and subdomains, including the primary domain, is redirected via htaccess 301s to their own subdirectories… >>> More
mod evasive not working properly on ubuntu 10.04

as seen on Server Fault - Search for 'Server Fault'
I have an ubuntu 10.04 server where I installed mod_evasive using apt-get install libapache2-mod-evasive I already tried several configurations, the result stays the same. The blocking does work, but randomly. I tried with low limis and long blocking periods as well as short limits. The behaviour… >>> More
Cross-domain jQuery using YQL gives robots.txt error

as seen on Stack Overflow - Search for 'Stack Overflow'
On the page http://qxlapps.dk/test.htm I am trying to perform an Ajax load from another domain, qxlapp.dk. I am using James Padolsey's xdomainajax.js plugin from: http://james.padolsey.com/javascript/cross-domain-requests-with-jquery/ When I open my test page, I get no output, but FireBug shows… >>> More
Asterisk in robots.txt

as seen on Stack Overflow - Search for 'Stack Overflow'
Wondering if following will work for google in robots.txt Disallow: /*.action I need to exclude all urls ending with .action. Is this correct? >>> More
SEO chaos from changing robots.txt file in Wordpress site

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi there, I recently edited the robots.txt file in my site using a wordpress plugin. However, since i did this, google seems to have removed my site from their search page. I'd appreciate if I could get an expert opinion on why this is so, and a possible solution. I'd initially done it to increase… >>> More

Related posts about web-crawlers

Discover How Affiliates Marketers Can Optimize Their Blog For Search Engine Web Crawlers - #1

as seen on Ezine Articles - Search for 'Ezine Articles'
Search Engine Optimization is a FREE way of internet marketing that all internet marketers should take full advantage of it. It is all about building links to your websites or blogs. There are 2 ways of doing it. >>> More
How do web crawlers affect site statistics?

as seen on Stack Overflow - Search for 'Stack Overflow'
What are ways in which web crawlers (both from search engines and non-search engines) could affect site statistics (e.g., when doing AB-testing different page variations)? And what are ways to take care of these problems? For example: Do a lot of people writing web crawlers often delete their cookies… >>> More
Is there a list of known web crawlers?

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm trying to get accurate download numbers for some files on a web server. I look at the user agents and some are clearly bots or web crawlers, but many for many I'm not sure, they may or may not be a web crawler and they are causing many downloads so it's important for me to know. Is there somewhere… >>> More
Detecting 'stealth' web-crawlers

as seen on Stack Overflow - Search for 'Stack Overflow'
What options are there to detect web-crawlers that do not want to be detected? (I know that listing detection techniques will allow the smart stealth-crawler programmer to make a better spider, but I do not think that we will ever be able to block smart stealth-crawlers anyway, only the ones that… >>> More
How to best develop web crawlers

as seen on Stack Overflow - Search for 'Stack Overflow'
Heyall, I am used to create some crawlers to compile information and as I come to a website I need the info I start a new crawler specific for that site, using shell scripts most of the time and sometime PHP. The way I do is with a simple for to iterate for the page list, a wget do download it and… >>> More