Can I use a Google Appliance/Mini to crawl and index sites I don't own?

Posted by SkippyFire on Server Fault See other posts from Server Fault or by SkippyFire
Published on 2010-07-23T18:55:31Z Indexed on 2011/01/05 22:55 UTC
Read the original article Hit count: 285

Maybe this is a stupid question, but...

I am working with this company and they said they needed to get "permission" to crawl other people's sites. They have a Google Search Appliance And some Google Minis and want to point them at other sites to aggregate content. The end result will be something like a targeted search engine. (All the indexed sites relate to a specific topic)

The only thing they will be doing is:

  1. Indexing Content from the other sites/domains
  2. Providing search functionality on their own site that searches the indexed content (like Google, displaying summaries and not the full content)
  3. The search results will provide links back to the original content

Their intent is not malicious in nature, and is to provide a single site/resource for people to reference on their given topic.

Is there anything illegal or fishy about this process?

© Server Fault or respective owner

Related posts about search

Related posts about indexing