Scraping &#151 character (long dash) error in Nokogiri

Posted by DavidP6 on Stack Overflow See other posts from Stack Overflow or by DavidP6
Published on 2010-05-12T18:43:19Z Indexed on 2010/05/12 18:44 UTC
Read the original article Hit count: 264

Filed under:

nokogiri

|

screen-scraping

I having trouble scraping a certain long dash that is encoded as — ; on the Time magazine site. It looks like this: —. It works fine when this dash is encoded as mdash, but when the problem dash is scraped, it is returned as unknown characters. I am using Nokogiri and am wondering if I have to use some sort of special encoding? The page says it is encoded with UTF-8.

© Stack Overflow or respective owner

Related posts about nokogiri

Nokogiri pull parser (Nokogiri::XML::Reader) issue with self closing tag

as seen on Stack Overflow - Search for 'Stack Overflow'
I have a huge XML(400MB) containing products. Using a DOM parser is therefore excluded, so i tried to parse and process it using a pull parser. Below is a snippet from the each_product(&block) method where i iterate over the product list. Basically, using a stack, i transform each <product>… >>> More
Can Nokogiri Parse a Nokogiri Object?

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm trying to parse a part of the result of a parse, like this: queries = @doc.xpath("//spectrum_query") queries.each do |query| hits = query.xpath("//search_hit") end But instead of Nokogiri only searching for search_hits inside the query, it searches for search_hits in the whole document… >>> More
webrat, rspec, nokogiri segfault

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm getting a segfault in nokogiri (1.4.1) run (under cucumber 0.6.1/webrat 0.7.0/rspec 1.3.x) response.should have_selector("div", :class => "fieldWithErrors") and the div in the page is actually <div class="fieldWithErrors validation_error"> stuff </div> Everything runs fine… >>> More
Leopard => Snow Leopard architecture woes with nokogiri / rails

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm confused. It's a regular state of affairs for me but specifically in this case I felt I could reach out to fellow stackoverflowers (that is, stackoverflow-ers, not stackover-flowers). uname -a Darwin macbookpro 10.3.0 Darwin Kernel Version 10.3.0: Fri Feb 26 11:58:09 PST 2010; root:xnu-1504.3… >>> More
How to access attributes using Nokogiri

as seen on Stack Overflow - Search for 'Stack Overflow'
I have a simple task of accessing the values of some attributes. Below is a simple script that uses Nokogiri::XML::Builder to create a simple xml doc. require 'nokogiri' builder = Nokogiri::XML::Builder.new(:encoding => 'UTF-8') do |xml| xml.Placement(:messageId => "392847-039820-938777"… >>> More

Related posts about screen-scraping

PHP Screen Scraping Class

as seen on Bradino - Search for 'Bradino'
After some positive feedback I have decided to continue to develop the PHP Screen Scraping class. This post will server as the permanent home for the class. Download PHP Screen Scraping Class Updates 20009-07-30 Added setHeader() function >>> More
Screen scraping over SSL with .NET

as seen on Stack Overflow - Search for 'Stack Overflow'
What solutions exist for screen scraping a site over SSL for use with .NET? My use case is that I need to login to a partner website (https), navigate through a dynamic hierarchy, and download a zipped file of reports. I certainly could use other screen scrapers if there are no good viable options… >>> More
looking for alternative to Webzinc .NET , screen scraping, web automation library for .net

as seen on Stack Overflow - Search for 'Stack Overflow'
i came across this .net library http://www.webzinc.com/online/faq.aspx however, i was wondering if there was a free alternative out there ? >>> More
Screen-scraping of a secure page of any site on https:// with asp.net in C#

as seen on Stack Overflow - Search for 'Stack Overflow'
I've done site scraping of secure page of any site on http:// but when I am trying to scrap any site on https:// then i always scrape the login page not secure page. Please advice what should i do for scraping a secure page of any site on https://. >>> More
How different is mashup from screenscraping and consuming webservices

as seen on Stack Overflow - Search for 'Stack Overflow'
From what I understand, Mashup is aggregating data from separate sources and providing a single view. How different is mashup when compared to screenscraping or using webservices to get data from external sources? >>> More