Difficulty screen scraping http://www.momondo.com using nokogiri
- by Khai Kiong
I have some difficulty to extract the total price (css selector = '.total') from the flight result.
http://www.momondo.com/multicity/?Search=true&TripType=oneway&SegNo=1&SO0=KUL&SD0=KBR&SDP0=31-12-2012&AD=2&CA=0,0&DO=false&NA=false#Search=true&TripType=oneway&SegNo=1&SO0=KUL&SD0=KBR&SDP0=31-12-2012&AD=2&CA=0,0&DO=false&NA=false
I get the error "undefined method `text' for nil:NilClass nokogiri ".
My code
desc "Fetch product prices"
task :fetch_details => :environment do
require 'nokogiri'
require 'open-uri'
include ERB::Util
OneWayFlight.find_all_by_money(nil).each do |flight|
url = "http://www.momondo.com/multicity/Search=true&TripType=oneway&SegNo=1&SO0=KUL&SD0=KBR&SDP0=31-12-2012&AD=2&CA=0,0&DO=false&NA=false#Search=true&TripType=oneway&SegNo=1&SO0=KUL&SD0=KBR&SDP0=31-12-2012&AD=2&CA=0,0&DO=false&NA=false"
doc = Nokogiri::HTML(open(url))
price = doc.at_css(".total").text[/[0-9\.]+/]
flight.update_attribute(:price, price)
end
end
