Can YQL parse web sites requiring cookie-based authentication?

Posted by user249488 on Stack Overflow See other posts from Stack Overflow or by user249488
Published on 2010-01-13T03:43:39Z Indexed on 2010/03/20 18:01 UTC
Read the original article Hit count: 449

Filed under:

yql

|

yahoo

|

xpath

|

http

First, my use case:

I'm trying to use YQL's built in XPATH capabilities to scrape content from Yahoo! Fantasy Sports. It uses some sort of cookie-based authentication scheme. Basically, the sequence is:

1) Do an HTTP GET on the Yahoo! Login page 2) Parse the hidden inputs from the response and do an HTTP PUT with your Yahoo! Login on the form URL 3) Use the cookies returned from step 2 to GET any of the Fantasy Sports! websites that you have access to

My question is, does YQL support doing this to scrape data? The only authentication based examples I've seen use OAuth, but I haven't seen any examples of using YQL to parse websites with cookie-based authentication schemes

© Stack Overflow or respective owner

Related posts about yql

Cross domain AJAX requests using JQuery and YQL

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi , I need to use YQL (Yahoo Query Language) to perform a cross domain ajax request then bind the XML response into a JSON object and retrieve a value from the XML. The link shows how this is done using the YQL service as a proxy for cross domain requests: http://ajaxian.com/archives/using-yql-as-a-proxy-for-cross-domain-ajax For… >>> More
How to implement YQL paging?

as seen on Stack Overflow - Search for 'Stack Overflow'
I've read the YQL guide, and I keep reviewing http://developer.yahoo.com/yql/guide/yql-o...entables-paging and I have been looking at a few examples, but I'm still left pretty confused how YQL paging works. The problem that I am trying to tackle is creating a YQL open data table for the Mozilla… >>> More
Query a location's WOEID using YQL and javascript/jquery

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi , I need to query a locations WOEID and grab the WOEID value from the xml returned. So the user would type e.g. London, UK and I need to load the query as below: http://query.yahooapis.com/v1/public/yql?q=select%20woeid%20from%20geo.places%20where%20text%20%3D%20%22London%2C%20UK%2C%20UK%22&format=xml After… >>> More
Cross domain AJAX requests using YQL

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi , I need to query a locations WOEID and grab the WOEID value from the xml returned. So the user would type e.g. London, UK and I need to load the query as below: http://query.yahooapis.com/v1/public/yql?q=select%20woeid%20from%20geo.places%20where%20text%20%3D%20%22London%2C%20UK%2C%20UK%22&format=xml After… >>> More
Automating scraping of table data to XML

as seen on Stack Overflow - Search for 'Stack Overflow'
Problem I have a YQL query result that I'm trying to get converted and sort into a clean XML file. Background Being the pains that they are, information from the World Cup isn't freely available in an easy to reuse format. So, after a bit of finessing with YQL I have managed to liberate the required… >>> More

Related posts about yahoo

Yahoo Store SEO - A Must For Yahoo Stores

as seen on Ezine Articles - Search for 'Ezine Articles'
Yahoo Store SEO is an essential activity for Yahoo Store owners as it helps in better rankings and more visits to the store. This article mentions some of the important aspects of Yahoo Store SEO for Yahoo Store owners. >>> More
Incorporate Professionalism in Your Yahoo Store With Yahoo Store Enhancements

as seen on Ezine Articles - Search for 'Ezine Articles'
Yahoo Store enhancements are an essential element for Yahoo Stores if you want to take your online business to the next level. This article explains about enhancements and describes some of the Yahoo enhancements popular today. >>> More
Access Delicious API via Yahoo ID without redirect to Yahoo authorizing page

as seen on Stack Overflow - Search for 'Stack Overflow'
To be more specific: My delicious account is a Yahoo ID so access via OAuth is required. It's for my use only to add/remove bookmarks. I have the password. It's a command line python program, not web-based. My question is: Is it possible to access my delicious account without being redirected… >>> More
Removing malware of a particular kind

as seen on Super User - Search for 'Super User'
I need to remove some malware from my computer. It is a trojan, and very annoying. It blocks access to Google and search sites. The trojan, with its name spelled out on each line cause it seems to block sites when i reference it in a url, is a r t (some text to mess it up) e m (more text i s First… >>> More
How to Block a HTTP Website along with Its All Subdomain using IPTABLE

as seen on Server Fault - Search for 'Server Fault'
I run a small HTTP web proxy site . We can not modify anything there in Proxy program. Few users mainly use Yahoo Web mail for Spamming and We need to block yahoo web mail access only ( complete yahoo website is also Ok) through our proxy . specially .mail.yahoo.com.. Like - we need to… >>> More