Server overhead caused by bots?
- by giuseppe
I have one customer website causing overhead (http://www.modacalcio.it/en/by-kind/football-boots.html).
With htop opened, I am trying navigate the website and the much load of the website is done by the ajax link being placed on the left side of the website.
The website is hosted by a VPS with 3 proc and 2GB RAM, with enough hard with disk space.
The real problem is that this website is new and not visited much.
From the http-status module I am seeing that the overhead is caused by bots (Google bots, Bing bots, hrefs checker and so on).
So I thought that's probably due to those spiders trying to crawl all those links at once - could this be causing this overhead?
I have also put rel="nofollow" in those links, but this doesn't keep the bots away.
Is there any way through code or Plesk to disable those links to those bots?