Scraping ASP.NET site with Ruby
Posted
by JillianK
on Stack Overflow
See other posts from Stack Overflow
or by JillianK
Published on 2010-05-12T21:36:50Z
Indexed on
2010/05/12
21:54 UTC
Read the original article
Hit count: 212
I would like to scrape the search results of this ASP.NET site using Ruby and preferably just using Hpricot (I cannot open an instance of Firefox): http://www.ngosinfo.gov.pk/SearchResults.aspx?name=&foa=0
However, I am having trouble figuring out how to go through each page of results. Basically, I need simulate clicking on links like these:
<a href="javascript:__doPostBack('ctl00$ContentPlaceHolder1$Pager1$2','')" class="blue_11" id="ctl00_ContentPlaceHolder1_Pager1">2</a>
<a href="javascript:__doPostBack('ctl00$ContentPlaceHolder1$Pager1$3','')" class="blue_11" id="ctl00_ContentPlaceHolder1_Pager1">3</a>
etc.
I tried using Net::HTTP to handle the post, but while that received the correct HTML, there were no search results (I'm probably not doing that correctly). In addition, the URL of the page does not contain any parameters indicating page, so it is not possible to force the results that way.
Any help would be greatly appreciated.
© Stack Overflow or respective owner