Nokogiri Doc Element Not Returning Correctly

Posted by TenJack on Stack Overflow See other posts from Stack Overflow or by TenJack
Published on 2010-04-18T23:30:54Z Indexed on 2010/04/18 23:33 UTC
Read the original article Hit count: 351

Filed under:

nokogiri

|

scraping

I am trying to scrape a wiktionary entry:

uri = URI.parse("http://en.wiktionary.org/wiki/" + CGI.escape('abjure'))
doc = Nokogiri::HTML(open(uri, 'User-Agent' => 'ruby'))

but the doc shows no elements for this word. The other words work fine and this word used to work. I have no idea what changed. Anyone see anything wrong with this?

© Stack Overflow or respective owner

Related posts about nokogiri

Nokogiri pull parser (Nokogiri::XML::Reader) issue with self closing tag

as seen on Stack Overflow - Search for 'Stack Overflow'
I have a huge XML(400MB) containing products. Using a DOM parser is therefore excluded, so i tried to parse and process it using a pull parser. Below is a snippet from the each_product(&block) method where i iterate over the product list. Basically, using a stack, i transform each <product>… >>> More
Can Nokogiri Parse a Nokogiri Object?

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm trying to parse a part of the result of a parse, like this: queries = @doc.xpath("//spectrum_query") queries.each do |query| hits = query.xpath("//search_hit") end But instead of Nokogiri only searching for search_hits inside the query, it searches for search_hits in the whole document… >>> More
webrat, rspec, nokogiri segfault

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm getting a segfault in nokogiri (1.4.1) run (under cucumber 0.6.1/webrat 0.7.0/rspec 1.3.x) response.should have_selector("div", :class => "fieldWithErrors") and the div in the page is actually <div class="fieldWithErrors validation_error"> stuff </div> Everything runs fine… >>> More
Leopard => Snow Leopard architecture woes with nokogiri / rails

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm confused. It's a regular state of affairs for me but specifically in this case I felt I could reach out to fellow stackoverflowers (that is, stackoverflow-ers, not stackover-flowers). uname -a Darwin macbookpro 10.3.0 Darwin Kernel Version 10.3.0: Fri Feb 26 11:58:09 PST 2010; root:xnu-1504.3… >>> More
How to access attributes using Nokogiri

as seen on Stack Overflow - Search for 'Stack Overflow'
I have a simple task of accessing the values of some attributes. Below is a simple script that uses Nokogiri::XML::Builder to create a simple xml doc. require 'nokogiri' builder = Nokogiri::XML::Builder.new(:encoding => 'UTF-8') do |xml| xml.Placement(:messageId => "392847-039820-938777"… >>> More

Related posts about scraping

Screen-scraping of a secure page of any site on https:// with asp.net in C#

as seen on Stack Overflow - Search for 'Stack Overflow'
I've done site scraping of secure page of any site on http:// but when I am trying to scrap any site on https:// then i always scrape the login page not secure page. Please advice what should i do for scraping a secure page of any site on https://. >>> More
looking for alternative to Webzinc .NET , screen scraping, web automation library for .net

as seen on Stack Overflow - Search for 'Stack Overflow'
i came across this .net library http://www.webzinc.com/online/faq.aspx however, i was wondering if there was a free alternative out there ? >>> More
PHP Screen Scraping Class

as seen on Bradino - Search for 'Bradino'
After some positive feedback I have decided to continue to develop the PHP Screen Scraping class. This post will server as the permanent home for the class. Download PHP Screen Scraping Class Updates 20009-07-30 Added setHeader() function >>> More
Alert Log Scraping with Oracle&#146;s ADRCI Utility

as seen on Internet.com - Search for 'Internet.com'
Oracles new ADR with command interface shows promise for database administrators who like to script their own solution for quickly scraping the alert log and automatically looking for errors. >>> More
Web scraping etiquette

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm considering writing a simple web scraping application to extract information from a website that does not seem to specifically prohibit this. I've checked for other alternatives (eg RSS, web service) to get this information, but there are none available at this stage. Despite this I've also… >>> More