How to create a web crawler/spider/robot?

Posted by Chris on Stack Overflow See other posts from Stack Overflow or by Chris
Published on 2010-03-21T23:52:50Z Indexed on 2010/03/22 0:01 UTC
Read the original article Hit count: 507

Filed under:
|
|

Is there a way to make a web robot like websiteoutlook.com does? I need something that searches the internet for URLs only...I don't need links, descriptions, etc.

What is the best way to do this without getting too technical? I guess it could even be a cronjob that runs a PHP script grabbing URLs from Google, or is there a better way?

A simple example or a link to more information would be much appreciated.

© Stack Overflow or respective owner

Related posts about web

Related posts about spider