Guidelines for good webcrawler 'Etiquette'
- by Harry
I'm building a search engine (for fun) and it has just struck me that potentially my little project might wreak havok by clicking on ads and all sorts of problems.
So what are the guidelines for good webcrawler 'Etiquette'?
Things that spring to mind:
Observe Robot.txt instructions
Limit the number of simultaneous requests to the same domain
Don't follow ad links?
Stopping the crawler from clicking on ads
- This one is particularly on my mind at the moment... how do i stop my bot from 'clicking' on ads? if it is going straight to the url in the ad is it counted as a click?