How can I use Perl to scrape a website that reveals its content with JavaScript?

Posted by AmbroseChapel on Stack Overflow See other posts from Stack Overflow or by AmbroseChapel
Published on 2010-04-24T09:21:54Z Indexed on 2010/04/25 15:03 UTC
Read the original article Hit count: 200

Filed under:

perl

|

ie

|

JavaScript

|

mechanize

|

Windows

I need to write a Perl script to scrape a website. The website can only be scraped with JavaScript, and the user is on Windows.

I got some way with Win32::IE::Mechanize on my work machine, which has IE6, but then I moved to my netbook which has IE8, and can't even get as far as fetching a simple page.

Is Win32::IE::Mechanize up to date with the latest versions of IE?

But, more to the point, given a recent WinXP machine, what's the quickest, easiest way to scrape a site which only reveals its content via JavaScript?

© Stack Overflow or respective owner

Related posts about perl

Munin on Centos 6 - missing perl MODULE_COMPAT_5.8.8

as seen on Server Fault - Search for 'Server Fault'
I'm trying to install Munin on a new VPS through yum install munin but I keep getting an error about a missing perl module: Requires: perl(:MODULE_COMPAT_5.8.8). This is the perl version currently installed: v5.10.1. I've searched all around and still haven't found a solution for this. Here's the… >>> More
Pain removing a perl rootkit

as seen on Server Fault - Search for 'Server Fault'
So, we host a geoservice webserver thing at the office. Someone apparently broke into this box (probably via ftp or ssh), and put some kind of irc-managed rootkit thing. Now I'm trying to clean the whole thing up, I found the process pid who tries to connect via irc, but i can't figure out who's… >>> More
How To Avoid a Perl script calling an Another Perl Script

as seen on Stack Overflow - Search for 'Stack Overflow'
Hello, i am calling a perl script client.pl from a main script to capture the output of client.pl in @output. is there anyway to avoid the use of these two files so i can use the output of client.pl in main.pl itself here is my code.... main.pl ======= my @output = readpipe("client.pl"); client… >>> More
Perl :how to sort dates in perl

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi, How can I sort the dates in perl. my @dates = ( "02/11/2009" , "12/20/2001" , "11/21/2010" ) ; I have above dates in my array . How can I sort those dates... ? My date format is dd/mm/YYYY. >>> More
please suggest a perl book exclusively for perl programs

as seen on Stack Overflow - Search for 'Stack Overflow'
I want tha name of a perl book for only PERL PROGRAMS. The reason behind is I want to improve my programming skill in perl >>> More

Related posts about ie

Excel document opens in IE 64, not in IE 32

as seen on Super User - Search for 'Super User'
Whenever I click on a hyperlink to a scrip that outputs an Excel 8 document, I get a prompt from IE to open the file or save-as. If I click open in IE 32 bit, the document opens in Excel (which is what I want). If I click open in the 64 bit version of IE, the document opens in the browser. How… >>> More
Unable to either locate any wireless networks nor even connect to wifi

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I'm new to Linux. I currently have installed ubuntu 12.10. I had a previous problem with my wireless card (see url to see previous problem : How to enable wireless in a Fujitsu LH532?). It now shows Connect to hidden network and create new wireless network but now unfortunately it simply cannot find… >>> More
Java Problem when i upgraded from IE 7 to IE 8

as seen on Server Fault - Search for 'Server Fault'
I face problem with upgrading IE8 related to JAVA. The problem only with IE8, IE7 is OK and I have the latest Java version. After I upgrade the IE8, the graphics which has dynamics reading from the server is not updating and it shows NaN. I tried the following : - upgrade the server win2003 with latest… >>> More
Why version of chrome does not matter much more then firefox and firefox does not matter much as IE

as seen on Geeks with Blogs - Search for 'Geeks with Blogs'
Everything not perfect. in software the software make and growth by user feedback like what user expected from the software and want in next version of software. In a chrome Event i hear about the Chromium. you can find some interesting things here Video 1 Video 2 come to the point. when i hear… >>> More
Problem display in IE 7 & IE 6 - simplemodal-1.3.4 and Jquery 1.4.2

as seen on Stack Overflow - Search for 'Stack Overflow'
I have a link, after click this link, a modal was displayed. I used ModalDialog with code: $(document).ready(function() { //linkTTT is link id $("a#linkTTT").click(function() { //content is id of div that contains content $("#content").modal({ onOpen: function(dialog)… >>> More