Googlebot fetches my pages very frequent, rel-nofollow, meta-noindex or robots.txt-disallow

Posted by trante on Pro Webmasters See other posts from Pro Webmasters or by trante
Published on 2012-12-18T20:08:28Z Indexed on 2012/12/18 23:13 UTC
Read the original article Hit count: 463

Filed under:

robots.txt

|

googlebot

|

serps

|

nofollow

|

noindex

Googlebot fetches pages in my site very frequently. And this slowens my website. I don't want Googlebot to crawl too frequent.

I decreased crawl rate from Google webmaster tools. But I'm supposing to use these three tools:

Adding rel="nofollow" to my inner pages. So Googlebot won't crawl and index them.
Adding meta tag "noindex" so Google will remove this page from index and won't get it again.
Adding Disallow: /mySomeFolder/ to robots.txt and Googlebot won't crawl that pages.

I'm planning to use these methods for my 56.000 pages, except the most important 6-7 pages. Which method would you prefer and what would be disadvantages or advantages ? Or won't it change my website speed etc..

© Pro Webmasters or respective owner

Related posts about robots.txt

Robots.txt practices with .htaccess redirections (inherits)

as seen on Pro Webmasters - Search for 'Pro Webmasters'
I have a question regarding how to write robots.txt files for many domains and subdomains with redirects in place. We have a hosting account that enacts primary and add-on domains. All of our domains and subdomains, including the primary domain, is redirected via htaccess 301s to their own subdirectories… >>> More
mod evasive not working properly on ubuntu 10.04

as seen on Server Fault - Search for 'Server Fault'
I have an ubuntu 10.04 server where I installed mod_evasive using apt-get install libapache2-mod-evasive I already tried several configurations, the result stays the same. The blocking does work, but randomly. I tried with low limis and long blocking periods as well as short limits. The behaviour… >>> More
Cross-domain jQuery using YQL gives robots.txt error

as seen on Stack Overflow - Search for 'Stack Overflow'
On the page http://qxlapps.dk/test.htm I am trying to perform an Ajax load from another domain, qxlapp.dk. I am using James Padolsey's xdomainajax.js plugin from: http://james.padolsey.com/javascript/cross-domain-requests-with-jquery/ When I open my test page, I get no output, but FireBug shows… >>> More
Asterisk in robots.txt

as seen on Stack Overflow - Search for 'Stack Overflow'
Wondering if following will work for google in robots.txt Disallow: /*.action I need to exclude all urls ending with .action. Is this correct? >>> More
SEO chaos from changing robots.txt file in Wordpress site

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi there, I recently edited the robots.txt file in my site using a wordpress plugin. However, since i did this, google seems to have removed my site from their search page. I'd appreciate if I could get an expert opinion on why this is so, and a possible solution. I'd initially done it to increase… >>> More

Related posts about googlebot

Screen resolution of Googlebot mobile?

as seen on Pro Webmasters - Search for 'Pro Webmasters'
Does Googlebot-Mobile have a viewport resolution it sends across? If so, what is it? It's a general question with broad relevance, but I am asking with reference to responsive design: particularly when serving different image resolution to different viewports via JavaScript. While Googlebot has… >>> More
ASP.NET MVC GoogleBot Issues

as seen on Stack Overflow - Search for 'Stack Overflow'
I wrote a site using ASP.NET MVC, and although it is not completely SEO optimized at this point I figured it is a good start. What I'm finding is that when I use Google's Webmaster Tools to fetch my site (to see what a GoogleBot sees) it sees this. HTTP/1.1 200 OK Cache-Control: public, max-age=1148 Content-Type:… >>> More
Where Googlebot starts crawling?

as seen on Stack Overflow - Search for 'Stack Overflow'
Say if I register a domain and have developed it into a complete website. From where and how Googlebot knows that the new domain is up? Does it always start with the domain registry? If it starts with the registry, does that mean that anyone can have complete access to the registry's database? Thanks… >>> More
Googlebot fetches my pages very frequent, rel-nofollow, meta-noindex or robots.txt-disallow

as seen on Pro Webmasters - Search for 'Pro Webmasters'
Googlebot fetches pages in my site very frequently. And this slowens my website. I don't want Googlebot to crawl too frequent. I decreased crawl rate from Google webmaster tools. But I'm supposing to use these three tools: Adding rel="nofollow" to my inner pages. So Googlebot won't crawl and index… >>> More
Google-Bot fell in love with my 404-page

as seen on Server Fault - Search for 'Server Fault'
Every day my access-log looks kind of this: 66.249.78.140 - - [21/Oct/2013:14:37:00 +0200] "GET /robots.txt HTTP/1.1" 200 112 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 66.249.78.140 - - [21/Oct/2013:14:37:01 +0200] "GET /robots.txt HTTP/1.1" 200 112 "-" "Mozilla/5… >>> More