How to create a web crawler/spider/robot?
Posted
by Chris
on Stack Overflow
See other posts from Stack Overflow
or by Chris
Published on 2010-03-21T23:52:50Z
Indexed on
2010/03/22
0:01 UTC
Read the original article
Hit count: 507
Is there a way to make a web robot like websiteoutlook.com does? I need something that searches the internet for URLs only...I don't need links, descriptions, etc.
What is the best way to do this without getting too technical? I guess it could even be a cronjob that runs a PHP script grabbing URLs from Google, or is there a better way?
A simple example or a link to more information would be much appreciated.
© Stack Overflow or respective owner