How can I scrape specific data from a website

Posted by Stoney on Super User See other posts from Super User or by Stoney
Published on 2012-09-12T15:47:45Z Indexed on 2012/09/12 21:40 UTC
Read the original article Hit count: 301

Filed under:

microsoft-excel

|

wget

|

web-crawler

|

data-mining

I'm trying to scrape data from a website for research.

The urls are nicely organized in an example.com/x format, with x as an ascending number and all of the pages are structured in the same way. I just need to grab certain headings and a few numbers which are always in the same locations. I'll then need to get this data into structured form for analysis in Excel.

I have used wget before to download pages, but I can't figure out how to grab specific lines of text.

Excel has a feature to grab data from the web (Data->From Web) but from what I can see it only allows me to download tables. Unfortunately, the data I need is not in tables.

© Super User or respective owner

Related posts about microsoft-excel

Connecting to Microsoft Excel using Oracle Data Integrator

as seen on Oracle Blogs - Search for 'Oracle Blogs'
The posts in this series assume that you have some level of familiarity with ODI. The concepts of Topology, Data Server, Physical and Logical Architecture are used here assuming that you understand them in the context of ODI. If you need more details on these elements, please refer to the ODI Tutorial… >>> More
Connect Microsoft Excel To SQL Azure Database

as seen on Dot net Slackers - Search for 'Dot net Slackers'
A number of people have found my post about getting started with SQL Azure pretty useful. But, it's all worthless if it doesn't add up to user value. Database are like potential energy in physics-it's a promise that something could be put in motion. Users actually making decisions based on analysis… >>> More
How to code Microsoft Excel "Shift Cells Up" feature in SQL

as seen on Stack Overflow - Search for 'Stack Overflow'
Take a simple table like below: Column Headings: || Agent's Name || Time Logged In || Center || Row 1: Andrew || 12:30 PM || Home Base Row 2: Jeff || 7:00 AM || Virtual Base Row 3: Ryan || 6:30 PM || Test Base Now lets say that a single cell is deleted so the table now looks like this: Column… >>> More
a question on using Microsoft Excel to do data analysis

as seen on Stack Overflow - Search for 'Stack Overflow'
Hello everyone there, Me and my mates have been given an Excel spreasheet,which contains the data of the Census report. The column here corresponds to "Year" (from 2000-2010),the row means "City/Town",the cell is simply the population of a town at a particular year. We are up to doing some analysis… >>> More
Working with PivotTables in Excel

as seen on How to geek - Search for 'How to geek'
PivotTables are one of the most powerful features of Microsoft Excel. They allow large amounts of data to be analyzed and summarized in just a few mouse clicks. In this article, we explore PivotTables, understand what they are, and learn how to create and customize them. Note: This article… >>> More

Related posts about wget

Make wget not download files larger than X size

as seen on Super User - Search for 'Super User'
Okay, I give up. How do I size limit which files are downloaded, like say I don't want any files bigger than 2 MB? >>> More
How to start using Wget?

as seen on Super User - Search for 'Super User'
Please, forgive me for asking this question. Usually I would try to learn thisngs myself first before bothering others, but my situation is urgent - if I don't act now and don't download all my family pictures from this website, it will be closed in about two weeks from now and I will loose all of… >>> More
wget mirroring, subdomains and directories and cookies

as seen on Server Fault - Search for 'Server Fault'
Hi all, I have an account on a web page that is now "full" (ie I have used up all my allocated space) and I would like to make a mirror of that site. wget seems like the thing to use. The problem is that I would only like to mirror the sites the lie within this directory http://user.domain.com/room/2324343/transcript/… >>> More
How can I install things in Linux with *no yum* and *no wget*?

as seen on Super User - Search for 'Super User'
I'm a newbie to Linux (that mainly uses Windows and Mac OS X) needing some advice. I was trying to install git on a Linux machine today, and encountered some problems: Not knowing the version of the installed OS, I've opened the /proc/version file which said: Linux version 2.6.9-42.0.2.ELsmp (bhcompile@ls20-bc1-13… >>> More
Getting wget to dowload only files with specific name patterns

as seen on Stack Overflow - Search for 'Stack Overflow'
I want to use wget to DL some files. I want to DL only files whose name that fit a certain pattern, e.g. ???.txt and not any other *.txt files. Can this be done with wget? I could only find a way to --accept/--reject files based on the extension. Thanks! >>> More