Parsing a website

Posted by Phenom on Stack Overflow See other posts from Stack Overflow or by Phenom
Published on 2010-04-06T20:08:29Z Indexed on 2010/04/06 20:13 UTC
Read the original article Hit count: 288

Filed under:

website

|

parsing

|

programming-languages

I want to make a program that takes as user input a website address. The program then goes to that website, downloads it, and then parses the information inside. It outputs a new html file using the information from the website.

Specifically, what this program will do is take certain links from the website, and put the links in the output html file, and it will discard everything else.

Right now I just want to make it for websites that don't require a login, but later on I want to make it work for sites where you have to login, so it will have to be able to deal with cookies.

I'll also want to later on have the program be able to explore certain links and download information from those other sites.

What are the best programming languages or tools to do this?

© Stack Overflow or respective owner

Related posts about website

Need help redirecting http://website.com/ to http://www.website.com/

as seen on Stack Overflow - Search for 'Stack Overflow'
What I'm trying to do is to redirect my website visitors who enter "website.com" to "www.website.com". I would do this with a standard redirect, but I don't know how to make a site specific to WWW or non-WWW addresses. I see that Firefox thinks my site is clearly different at the WWW version, because… >>> More
Official BETA release of Developer IT

as seen on Developer IT - Search for 'Developer IT'
We finally did it It’s been a week since our first online publish and our indexer robot is going as well as the website. We already have reach more than 20,000 articles and it’s only the begining. Stay tune on http://www.developerit.com/ >>> More
Get Visitors to Your Website Using Website Building Tips

as seen on Ezine Articles - Search for 'Ezine Articles'
Before building a website there are many things that you need to take in mind. For example, how are you going to design your website, what is it going to cost you, how long will it take you to build, etc. All of these things mentioned are important aspects to consider when designing a website, but… >>> More
Website Optimization - Requirement of Every Website

as seen on Ezine Articles - Search for 'Ezine Articles'
Website optimization is now days a must for each and every website to be on the top at search engines result pages. Also it has become a vital method to drive traffic to the website. >>> More
Website Basics - Planning Your Website

as seen on Ezine Articles - Search for 'Ezine Articles'
A website begins with an idea. You probably have an idea for a site and that is the reason that you are exploring your options more. Or you may be looking to getting into Internet Marketing and need to know what is involved in getting a site up and running. >>> More

Related posts about parsing

Hot to fix nautilus desktop on linux mint

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
so I'm using Linux Mint 13 with Cinnamon and suddenly there are no icons on the desktop and the right click doesn't work, it's like the desktop doesn't start up at all, but the Cinnamon interface and everything else are working just fine. This happens only when I open the session with Cinnamon, if… >>> More
Is parsing JSON faster than parsing XML

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm creating a sophisticated JavaScript library for working with my company's server side framework. The server side framework encodes its data to a simple XML format. There's no fancy namespacing or anything like that. Ideally I'd like to parse all of the data in the browser as JSON. However, if… >>> More
Looking for a tutorial on Recursive Descent Parsing.

as seen on Stack Overflow - Search for 'Stack Overflow'
I am trying to parse some data to no success. Can anyone recommend a good introduction with a lot of examples to Recursive Descent Parsing? I haven't been able to find any. >>> More
Parsing XML with Hpricot, a Gem of a Ruby Gem

as seen on Internet.com - Search for 'Internet.com'
Need to parse complex XML documents but don't know where to begin? Leave the task to Ruby's powerful Hpricot library. >>> More
Parsing scripts that use curly braces

as seen on Programmers - Search for 'Programmers'
To get an idea of what I'm doing, I am writing a python parser that will parse directx .x text files. The problem I have deals with how the files are formatted. Although I'm writing it in python, I'm looking for general algorithms for dealing with this sort of parsing. .x files define data using… >>> More