Scraping ASP.NET site with Ruby

Posted by JillianK on Stack Overflow See other posts from Stack Overflow or by JillianK
Published on 2010-05-12T21:36:50Z Indexed on 2010/05/12 21:54 UTC
Read the original article Hit count: 212

Filed under:
|
|

I would like to scrape the search results of this ASP.NET site using Ruby and preferably just using Hpricot (I cannot open an instance of Firefox): http://www.ngosinfo.gov.pk/SearchResults.aspx?name=&foa=0

However, I am having trouble figuring out how to go through each page of results. Basically, I need simulate clicking on links like these:

<a href="javascript:__doPostBack('ctl00$ContentPlaceHolder1$Pager1$2','')" class="blue_11" id="ctl00_ContentPlaceHolder1_Pager1">2</a>
<a href="javascript:__doPostBack('ctl00$ContentPlaceHolder1$Pager1$3','')" class="blue_11" id="ctl00_ContentPlaceHolder1_Pager1">3</a>

etc.

I tried using Net::HTTP to handle the post, but while that received the correct HTML, there were no search results (I'm probably not doing that correctly). In addition, the URL of the page does not contain any parameters indicating page, so it is not possible to force the results that way.

Any help would be greatly appreciated.

© Stack Overflow or respective owner

Related posts about screen-scraping

Related posts about ASP.NET