Looking for a recommendation of a good tutorial on best practices for a web scraping project?

Posted by in bruges on Stack Overflow See other posts from Stack Overflow or by in bruges
Published on 2009-03-26T05:31:24Z Indexed on 2010/03/24 22:43 UTC
Read the original article Hit count: 285

Filed under:

hpricot

|

beautifulsoup

|

screen-scraping

|

ruby

|

python

I need to do a fairly extensive project involving web scraping and am considering using Hpricot or Beautiful Soup (i.e. Ruby or Python). Has anyone come across a tutorial that they thought was particularly good on this subject that would help me start the project off on the right foot?

© Stack Overflow or respective owner

Related posts about hpricot

how to translate this hpricot code to nokogiri ?

as seen on Stack Overflow - Search for 'Stack Overflow'
Hpricot(html).inner_text.gsub("\r"," ").gsub("\n"," ").split(" ").join(" ") hpricot = Hpricot(html) hpricot.search("script").remove hpricot.search("link").remove hpricot.search("meta").remove hpricot.search("style").remove found it on http://www.savedmyday.com/2008/04/25/how-to-extract-text-from-html-using-rubyhpricot… >>> More
How to get an element using inner text (Watir, Nokogir, Hpricot)

as seen on Stack Overflow - Search for 'Stack Overflow'
I have been expeirmenting with Watir, Nokogir and Hpricot. All of these use top-down approach which is my problem. i.e. they use element type to search element. I want to find out the element using the text without knowing element type. e.g. <element1> <element2> Text2 </element2> … >>> More
hpricot using java?

as seen on Stack Overflow - Search for 'Stack Overflow'
I've just noticed that most of hpricot code is written in java... I heard that JRuby performed a lot better than native ruby when processing regular expression. Is maybe the java classes just activated if JRuby or Java is installed and the ruby used if these are not found? It's something puzzling… >>> More
hpricot segfault?

as seen on Stack Overflow - Search for 'Stack Overflow'
Any idea why hpricot might segfault on this page? trial_url = 'http://www.controlled-trials.com/ISRCTN56071145/' doc = Hpricot(open(trial_url)) produces: /Users/ap257/.gem/ruby/1.8/gems/hpricot-0.8.2/lib/hpricot/parse.rb:33: [BUG] Segmentation fault ruby 1.8.7 (2009-06-08 patchlevel 173) [universal-darwin10… >>> More
hpricot throws exception when trying to parse url which has noscript tag

as seen on Stack Overflow - Search for 'Stack Overflow'
I use hpricot gem in ruby on rails to parse a webpage and extract the meta-tag contents. But if the website has a <noscrpit> tag just after the <head> tag it throws an exception Exception: undefined method `[]' for nil:NilClass I even tried to update the gem to the latest version. but… >>> More

Related posts about beautifulsoup

Getting BeautifulSoup to find a specific <p>

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm trying to put together a basic HTML scraper for a variety of scientific journal websites, specifically trying to get the abstract or introductory paragraph. The current journal I'm working on is Nature, and the article I've been using as my sample can be seen at http://www.nature.com/nature/journal/v463/n7284/abs/nature08715… >>> More
Trying to grab just absolute links from a webpage using BeautifulSoup

as seen on Stack Overflow - Search for 'Stack Overflow'
I am reading the contents of a webpage using BeautifulSoup. What I want is to just grab the <a href> that start with http://. I know in beautifulsoup you can search by the attributes. I guess I am just having a syntax issue. I would imagine it would go something like. page = urllib2.urlopen("http://www… >>> More
Extracting an attribute value with beautifulsoup

as seen on Stack Overflow - Search for 'Stack Overflow'
I am trying to extract the content of a single "value" attribute in a specific "input" tag on a webpage. I use the following code: import urllib f = urllib.urlopen("http://58.68.130.147") s = f.read() f.close() from BeautifulSoup import BeautifulStoneSoup soup = BeautifulStoneSoup(s) inputTag =… >>> More
beautifulsoup and mechanize to get ajax call result

as seen on Stack Overflow - Search for 'Stack Overflow'
hi im building a scraper using python 2.5 and beautifulsoup but im stuble upon a problem ... part of the web page is generating after user click on some button, whitch start an ajax request by calling specific javacsript function using proper parameters is there a way to simulate user interaction… >>> More
Python beautifulsoup trying to remove html tags 'span'

as seen on Stack Overflow - Search for 'Stack Overflow'
I am trying to remove [<span class="street-address"> 510 E Airline Way </span>] and I have used this clean function to remove the one that is in between < > def clean(val): if type(val) is not StringType: val = str(val) val = re.sub(r'<.*?>', ''… >>> More