Can YQL parse web sites requiring cookie-based authentication?
- by user249488
First, my use case:
I'm trying to use YQL's built in XPATH capabilities to scrape content from Yahoo! Fantasy Sports. It uses some sort of cookie-based authentication scheme. Basically, the sequence is:
1) Do an HTTP GET on the Yahoo! Login page
2) Parse the hidden inputs from the response and do an HTTP PUT with your Yahoo! Login on the form URL
3) Use the cookies returned from step 2 to GET any of the Fantasy Sports! websites that you have access to
My question is, does YQL support doing this to scrape data? The only authentication based examples I've seen use OAuth, but I haven't seen any examples of using YQL to parse websites with cookie-based authentication schemes