Server overhead caused by bots?

Posted by giuseppe on Pro Webmasters See other posts from Pro Webmasters or by giuseppe
Published on 2013-08-01T11:04:12Z Indexed on 2013/08/02 16:06 UTC
Read the original article Hit count: 318

Filed under:

googlebot

|

vps

|

plesk

|

bingbot

I have one customer website causing overhead (http://www.modacalcio.it/en/by-kind/football-boots.html).

With htop opened, I am trying navigate the website and the much load of the website is done by the ajax link being placed on the left side of the website.

The website is hosted by a VPS with 3 proc and 2GB RAM, with enough hard with disk space.

The real problem is that this website is new and not visited much.

From the http-status module I am seeing that the overhead is caused by bots (Google bots, Bing bots, hrefs checker and so on).

So I thought that's probably due to those spiders trying to crawl all those links at once - could this be causing this overhead?

I have also put rel="nofollow" in those links, but this doesn't keep the bots away.

Is there any way through code or Plesk to disable those links to those bots?

© Pro Webmasters or respective owner

Related posts about googlebot

Screen resolution of Googlebot mobile?

as seen on Pro Webmasters - Search for 'Pro Webmasters'
Does Googlebot-Mobile have a viewport resolution it sends across? If so, what is it? It's a general question with broad relevance, but I am asking with reference to responsive design: particularly when serving different image resolution to different viewports via JavaScript. While Googlebot has… >>> More
ASP.NET MVC GoogleBot Issues

as seen on Stack Overflow - Search for 'Stack Overflow'
I wrote a site using ASP.NET MVC, and although it is not completely SEO optimized at this point I figured it is a good start. What I'm finding is that when I use Google's Webmaster Tools to fetch my site (to see what a GoogleBot sees) it sees this. HTTP/1.1 200 OK Cache-Control: public, max-age=1148 Content-Type:… >>> More
Where Googlebot starts crawling?

as seen on Stack Overflow - Search for 'Stack Overflow'
Say if I register a domain and have developed it into a complete website. From where and how Googlebot knows that the new domain is up? Does it always start with the domain registry? If it starts with the registry, does that mean that anyone can have complete access to the registry's database? Thanks… >>> More
Googlebot fetches my pages very frequent, rel-nofollow, meta-noindex or robots.txt-disallow

as seen on Pro Webmasters - Search for 'Pro Webmasters'
Googlebot fetches pages in my site very frequently. And this slowens my website. I don't want Googlebot to crawl too frequent. I decreased crawl rate from Google webmaster tools. But I'm supposing to use these three tools: Adding rel="nofollow" to my inner pages. So Googlebot won't crawl and index… >>> More
Google-Bot fell in love with my 404-page

as seen on Server Fault - Search for 'Server Fault'
Every day my access-log looks kind of this: 66.249.78.140 - - [21/Oct/2013:14:37:00 +0200] "GET /robots.txt HTTP/1.1" 200 112 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 66.249.78.140 - - [21/Oct/2013:14:37:01 +0200] "GET /robots.txt HTTP/1.1" 200 112 "-" "Mozilla/5… >>> More

Related posts about vps

Primary domain in vps in vps has been deactivated

as seen on Pro Webmasters - Search for 'Pro Webmasters'
This is my scenario: I have a vps with two domains (example1.com, example2.com). When I started with this vps I set example1.com as primary domain and the nameserver were configured with the pattern ns1.example.com, ns2.example1.com. The domains were brought in name.com. Across the time, I usually… >>> More
Several IPs for my VPS

as seen on Server Fault - Search for 'Server Fault'
I bought vps on santrex.net but can't receive any reply from support. My Problem: I have 5 ip but it pings only 1!!! I can't setup DNS because I need 2 ip minimum . Could you help me to activate other my IPs? root@spnova:~# ifconfig eth0 Link encap:Ethernet HWaddr aa:00:b9:4f:19:01 … >>> More
OS for Virtual Private Server (VPS)

as seen on Server Fault - Search for 'Server Fault'
Hi guys! I'm new to VPS managing and I need to chose which OS to install on my VPS. I have the following alternatives: CentOS 5.2 Ubuntu 9.10 Ubuntu Server edition 8.10 Debian 5.0 Lenny Debian 4.0 Etch Gentoo Minimal 10.0 I tried to install couple of them, I've also installed Webmin and doesn't… >>> More
Can't connect to website after altering IPTables

as seen on Server Fault - Search for 'Server Fault'
I attempted to open up a port on my VPS, but I can't connect to my website after opening up that port. Below are the commands I issued to open the port. I didn't get an error or anything after setting this up. I just can't connect to my website after doing this. [root@vps ~]# iptables -F [root@vps… >>> More
The vps is running, but the domain name is down ?

as seen on Server Fault - Search for 'Server Fault'
Hi, i created a vps on vps.net and installed CentOS 5.4 x64 LAMP , i choosed the root password and the server start running, i added my domain name in the vps domain section, also i added the two name servers on where i bought the domain name. the problem is that i still can't access the domain… >>> More