How to write a crawler?
Posted
by Jason
on Stack Overflow
See other posts from Stack Overflow
or by Jason
Published on 2008-09-19T15:09:24Z
Indexed on
2010/05/22
0:30 UTC
Read the original article
Hit count: 385
Hi All,
I have had thoughts of trying to write a simple crawler that might crawl and produce a list of its findings for our NPO's websites and content.
Does anybody have any thoughts on how to do this? Where do you point the crawler to get started? How does it send back its findings and still keep crawling? How does it know what it finds, etc,etc.
Thanks! -Jason
© Stack Overflow or respective owner