How do travel search engines & aggregators get their source data?

Posted by Alex on Stack Overflow See other posts from Stack Overflow or by Alex
Published on 2009-10-03T13:50:32Z Indexed on 2010/06/15 16:02 UTC
Read the original article Hit count: 213

Filed under:

webcrawling

I'm tossing around a few ideas for travel search engines and I'm wondering how these sites get their source data. Do they scrape all the content from airline homepages? This seems like an enormous job given the number of airlines etc out there.

Is there some API or web service standard that every airline conforms too?

Am I going to have to negotiate access to the data APIs for every airline I wish to index?

Related posts about webcrawling

Asynchronous Webcrawling F#, something wrong ?

as seen on Stack Overflow - Search for 'Stack Overflow'
Not quite sure if it is ok to do this but, my question is: Is there something wrong with my code ? It doesn't go as fast as I would like, and since I am using lots of async workflows maybe I am doing something wrong. The goal here is to build something that can crawl 20 000 pages in less than an hour… >>> More
WebCrawling Dynamic Links

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi Everyone, Anybody has any idea on crawling websites that have dynamic pages/queries? I mean if I click a certain link, it has different values every I try to reload it in a web browser. Now my webcrawler could not download the contents of these pages. Please advise. >>> More
Crawling engine architecture - Java/ Perl integration

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi all, I am looking to develop a management and administration solution around our webcrawling perl scripts. Basically, right now our scripts are saved in SVN and are manually kicked off by SysAdmin/devs etc. Everytime we need to retrieve data from new sources we have to create a ticket with business… >>> More
Building an automatic web crawler

as seen on Stack Overflow - Search for 'Stack Overflow'
I am building a web application crawler that's meant not only to find all the links or pages in a web application, but also perform all the allowed actions in the app (such as pushing buttons, filling forms, notice changes in the DOM even if they did not trigger a request etc.) Basically, this is… >>> More
Getting web page after calling DownloadStringAsync()?

as seen on Stack Overflow - Search for 'Stack Overflow'
Hello I don't know enough about VB.Net yet to use the richer HttpWebRequest class, so I figured I'd use the simpler WebClient class to download web pages asynchronously (to avoid freezing the UI). However, how can the asynchronous event handler actually return the web page to the calling routine… >>> More

Developer IT

How do travel search engines & aggregators get their source data? - Developer IT

How do travel search engines & aggregators get their source data?

webcrawling

Related posts about webcrawling

Asynchronous Webcrawling F#, something wrong ?

WebCrawling Dynamic Links

Crawling engine architecture - Java/ Perl integration

Building an automatic web crawler

Getting web page after calling DownloadStringAsync()?

Categories cloud