Do or can robots cause considerable performance issues?
Posted
by
Anicho
on Pro Webmasters
See other posts from Pro Webmasters
or by Anicho
Published on 2012-07-05T13:55:14Z
Indexed on
2012/07/05
15:23 UTC
Read the original article
Hit count: 455
So the question in the title is exactly what I am trying to find out.
My case is:
At work we are in a discussion with team members who seem to think bots will cause us problems relating to performance when running on our services website.
Out setup:
Lets say I have site www.mysite.co.uk this is a shop window to our online services which sit on www.mysiteonline.co.uk.
When people search in google for mysite they see mysiteonline.co.uk as well as mysite.co.uk.
Cases against stopping bots crawling:
- We don't store gb's of data publicly available on the web
- Most friendly bots, if they were to cause issues would have done so already
- In our instance the bots can't crawl the site because it requires username & password
- Stopping bots with robot .txt causes an issue with seo (ref.1)
- If it was a malicious bot, it would ignore robot.txt or meta tags anyway
Ref 1. If we were to block mysiteonline.co.uk from having robots crawl this will affect seo rankings and make it inconvenient for users who actively search for mysite to find mysiteonline. Which we can prove is the case for a good portion of our users.
© Pro Webmasters or respective owner