Difficulty screen scraping http://www.momondo.com using nokogiri

Posted by Khai Kiong on Stack Overflow See other posts from Stack Overflow or by Khai Kiong
Published on 2012-11-25T14:09:50Z Indexed on 2012/11/25 17:04 UTC
Read the original article Hit count: 409

Filed under:

ruby-on-rails

|

nokogiri

I have some difficulty to extract the total price (css selector = '.total') from the flight result.

http://www.momondo.com/multicity/?Search=true&TripType=oneway&SegNo=1&SO0=KUL&SD0=KBR&SDP0=31-12-2012&AD=2&CA=0,0&DO=false&NA=false#Search=true&TripType=oneway&SegNo=1&SO0=KUL&SD0=KBR&SDP0=31-12-2012&AD=2&CA=0,0&DO=false&NA=false

I get the error "undefined method `text' for nil:NilClass nokogiri ".

My code

desc "Fetch product prices"
task :fetch_details => :environment do
  require 'nokogiri'
  require 'open-uri'
  include ERB::Util

  OneWayFlight.find_all_by_money(nil).each do |flight|
    url = "http://www.momondo.com/multicity/Search=true&TripType=oneway&SegNo=1&SO0=KUL&SD0=KBR&SDP0=31-12-2012&AD=2&CA=0,0&DO=false&NA=false#Search=true&TripType=oneway&SegNo=1&SO0=KUL&SD0=KBR&SDP0=31-12-2012&AD=2&CA=0,0&DO=false&NA=false"
    doc = Nokogiri::HTML(open(url))
    price = doc.at_css(".total").text[/[0-9\.]+/]
    flight.update_attribute(:price, price)
  end
end

© Stack Overflow or respective owner

Related posts about ruby-on-rails

Ruby on Rails - How can I start? [closed]

as seen on Programmers - Search for 'Programmers'
I have misconception in understanding the relationship between Ruby language and Ruby on Rails Framework. Because of 'I am an absolute beginner' in web development I have no idea if I have to grasp the fundamentals of Ruby before I go with Ruby on Rails! I also want to ask who is behind both Ruby… >>> More
DES3 decryption in Ruby on Rails

as seen on Stack Overflow - Search for 'Stack Overflow'
My RoR server receives a string, that was encrypted in C++ application using des3 with base64 encoding The cipher object is created so: cipher = OpenSSL::Cipher::Cipher::new("des3") cipher.key = key_str cipher.iv = iv_str key_str and iv_str: are string representations of key and initialization… >>> More
Ruby on rails: Image downloads with Authentication/Authorization/Time outs

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi Guys, I'm having few doubts on implementing file downloads. I'm creating an app where I use attachment_fu with Amazon s3 to upload files. Things are working pretty well so far on uploading side. Now its the time to start the file downloads. Here is what I need, a logged in user search and browse… >>> More
Ruby on Rails deployment, on "thin" server with lot of attachments

as seen on Stack Overflow - Search for 'Stack Overflow'
A lot of PDFs are stored inside MySQL as a BLOB field for each PDF file. The average file size is 500K each. The Rails app will stream the :binary data as file downloads, where there is a user click on the download link. Assume there is a maximum of 5 users downloading 5 PDFs concurrently, what… >>> More
Apply Behavior Driven Development to Ruby on Rails with Rspec

as seen on Internet.com - Search for 'Internet.com'
Applying the Behavior Driven Development (BDD) methodology to your Rails development takes you beyond traditional Test Driven Development. Learn how to do it with the Ruby-based BDD framework RSpec. >>> More

Related posts about nokogiri

Nokogiri pull parser (Nokogiri::XML::Reader) issue with self closing tag

as seen on Stack Overflow - Search for 'Stack Overflow'
I have a huge XML(400MB) containing products. Using a DOM parser is therefore excluded, so i tried to parse and process it using a pull parser. Below is a snippet from the each_product(&block) method where i iterate over the product list. Basically, using a stack, i transform each <product>… >>> More
Can Nokogiri Parse a Nokogiri Object?

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm trying to parse a part of the result of a parse, like this: queries = @doc.xpath("//spectrum_query") queries.each do |query| hits = query.xpath("//search_hit") end But instead of Nokogiri only searching for search_hits inside the query, it searches for search_hits in the whole document… >>> More
webrat, rspec, nokogiri segfault

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm getting a segfault in nokogiri (1.4.1) run (under cucumber 0.6.1/webrat 0.7.0/rspec 1.3.x) response.should have_selector("div", :class => "fieldWithErrors") and the div in the page is actually <div class="fieldWithErrors validation_error"> stuff </div> Everything runs fine… >>> More
Leopard => Snow Leopard architecture woes with nokogiri / rails

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm confused. It's a regular state of affairs for me but specifically in this case I felt I could reach out to fellow stackoverflowers (that is, stackoverflow-ers, not stackover-flowers). uname -a Darwin macbookpro 10.3.0 Darwin Kernel Version 10.3.0: Fri Feb 26 11:58:09 PST 2010; root:xnu-1504.3… >>> More
How to access attributes using Nokogiri

as seen on Stack Overflow - Search for 'Stack Overflow'
I have a simple task of accessing the values of some attributes. Below is a simple script that uses Nokogiri::XML::Builder to create a simple xml doc. require 'nokogiri' builder = Nokogiri::XML::Builder.new(:encoding => 'UTF-8') do |xml| xml.Placement(:messageId => "392847-039820-938777"… >>> More