Legality, terms of service for performing a web crawl
- by Berlin Brown
I was going to crawl a site for some research I was collecting. But, apparently the terms of service is quite clear on the topic. Is it illegal to now "follow" the terms of service. And what can the site normally do?
Here is an example clause in the TOS. Also, what about sites that don't provide this particular clause.
Restrictions: "use any robot, spider, site search application, or other automated device, process or means to access, retrieve, scrape, or index the site"
It is just research?
Edit: "OK, from the standpoint of designing an efficient crawler. Should I provide some form of natural language engine to read terms of service and then abide by them."