How to write a crawler?

Posted by Jason on Stack Overflow See other posts from Stack Overflow or by Jason
Published on 2008-09-19T15:09:24Z Indexed on 2010/05/22 0:30 UTC
Read the original article Hit count: 448

Filed under:

internet

|

crawler

Hi All,

I have had thoughts of trying to write a simple crawler that might crawl and produce a list of its findings for our NPO's websites and content.

Does anybody have any thoughts on how to do this? Where do you point the crawler to get started? How does it send back its findings and still keep crawling? How does it know what it finds, etc,etc.

Thanks! -Jason

© Stack Overflow or respective owner

Related posts about internet

Connect mobile phone to internet using desktop pc internet

as seen on Super User - Search for 'Super User'
is it possible to share my desktop pc internet to my mobile phone? im using HTC Hero and a wired internet connection on my desktop pc >>> More
T3 Internet - The Great Internet Network for your Call Center Company

as seen on Article City - Search for 'Article City'
The most beneficial internet connection to get for your contact center enterprise would be a T3 Internetconnection. You are able to assume so several fantastic points from this type of connection, co... [Author: Van Theodorou - Computers and Internet - May 29, 2010] >>> More
Windows 8: Is Internet Explorer10 Metro mode disabled? Here is the fix.

as seen on Tech Dreams - Search for 'Tech Dreams'
Are you having issues with Internet Explorer 10 on Windows 8? Is Internet Explorer 10 application not launching in Metro mode and always launching in Desktop mode? Continue reading the post to understand and fix the problem. When Internet Explorer 10 does not launch in Metro mode? On Windows 8 when… >>> More
Cannot Connect to internet in windows 7 (No Internet Connection)

as seen on Super User - Search for 'Super User'
I have an airport express and Win 7 pro. My router is a Trendnet TEW-423PI G. The problem is everytime i try to connect it says it is unsuccessful and can't find ip address. I have tried everything. WPA2, WPA, WEP, Messing with the registry, Manually Inputting the ip address, Reinstalling drivers… >>> More
IE7 Internet Explorer cannot open the internet site, operation aborted

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi, when i test this page http://www.catalogues4u.com.au/ViewCategory.aspx?catID=119 im getting the above error. to replicate this issue visit the above page in ie7 and you will get the prompt. i did some search but couldnt find a way to resolve this issue. any help is apprecated. THanks, Aneef >>> More

Related posts about crawler

Site crawler/spider that tosses results into mysql

as seen on Server Fault - Search for 'Server Fault'
It's been suggested that we use mysql for our site's search as it'd be running on the same server that hosts our web server (nginx) and our db (mysql). Since not all of our pages are created from the database, it's been suggested that we have a crawler that can crawl the site, and toss the page url… >>> More
Remove subdomain from Google Crawler

as seen on Server Fault - Search for 'Server Fault'
Hi all, I recently removed a sub-domain from my domain so I just have 1 website to manage. However, if I do a google search, my old domain is still there, I removed the sub-domain well over a week ago and if you try to access the domain directly, you will get an error saying the website can not… >>> More
Is there an automated way to take site inventory?

as seen on Pro Webmasters - Search for 'Pro Webmasters'
Is there a way to take site inventory using a crawler program that checks either the sources of images for specific servers that serve ads, or, that the crawler looks at a page for specific (html5?) tags like <aside> or some other tag to count the inventory of ad spaces available on a site?… >>> More
Building an automatic web crawler

as seen on Stack Overflow - Search for 'Stack Overflow'
I am building a web application crawler that's meant not only to find all the links or pages in a web application, but also perform all the allowed actions in the app (such as pushing buttons, filling forms, notice changes in the DOM even if they did not trigger a request etc.) Basically, this is… >>> More
What is a good Java crawler library?

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi, I am about to develop a crawler in Java but don't feel like reinventing the wheel. A quick Google search gives a whole bunch of Java libraries to build a web crawler. Besides that Nutch is of course a very robust package but seems a bit too advanced for my needs. I only need to crawl a handful… >>> More