How to crawl a webPage with dynamic content added by javascript

Posted by blunderboy on Pro Webmasters See other posts from Pro Webmasters or by blunderboy
Published on 2012-08-31T05:11:42Z Indexed on 2012/08/31 9:51 UTC
Read the original article Hit count: 280

Filed under:

JavaScript

|

crawling

I guess there is a news that Google bots have the capability to understand our javascript code. It means this is possible to fully crawl a webpage which has lazy loading feature enabled. I am using Apache Nutch to crawl websites but I don't think it has the capability to fetch the URLs being injected in HTML page by javascript when the page is scrolled down. I see a lot of websites doing lazy loading for performance issue. So Can somebody please explain me how can i crawl the data which comes in HTML page on lazy load. (On scrolling the page down).

© Pro Webmasters or respective owner

Related posts about JavaScript

CHAT ROOMs 7 by 6

as seen on Stack Overflow - Search for 'Stack Overflow'
I am looking for chatroom on one page with 7 loggedin users and 6+rows for say 42 users.these users will keep on adding wthnew users.Need urgent help.A PRETTY UNUSUAL Q FOR MOST OF U.What is MORE REQ new features: Usernames are unique to users currently chatting You can see a "currently chatting"… >>> More
Integrating JavaScript Unit Tests with Visual Studio

as seen on Stephen Walter - Search for 'Stephen Walter'
Modern ASP.NET web applications take full advantage of client-side JavaScript to provide better interactivity and responsiveness. If you are building an ASP.NET application in the right way, you quickly end up with lots and lots of JavaScript code. When writing server code, you should be writing… >>> More
Has Javascript developed beyond what it was originally designed to do?

as seen on Programmers - Search for 'Programmers'
I've been talking with a friend about the purpose of Javascript, when and how it should be used, etc. He quoted that: JavaScript was designed to add interactivity to HTML pages [...] JavaScript gives HTML designers a programming tool HTML authors are normally not programmers… >>> More
PHP, javascript, single quote problems with IE when passing variable from ajax post to javascript fu

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi! I have been trying to get this to work for a while, and I suspect there's an easy solution that I just can't find. My head feels like jelly and I would really appreciate any help. My main page.php makes a .post() to backend.php and fetches a list of cities which it echoes in the form of: <li… >>> More
Javascript in XSL that is loaded by Javascript

as seen on Stack Overflow - Search for 'Stack Overflow'
Is there anyway to have javascript run when a XSL sheet has been applied to an XML file by Javascript? I am using a JQuery plugin to apply the sheet to the xml but the javascript that is located inside of the XSL file will not run. I put the Javascript at the bottom of the file and it still does… >>> More

Related posts about crawling

https & ajax crawling

as seen on Pro Webmasters - Search for 'Pro Webmasters'
We made on our webpage https://www.1point618.com a transition to ssl and now we using nearly entirely ajax to load the content. Therefore all urls of existing pages have changed. We used the 301 redirect as recommended, also we have implemented google's specification that the webpage is still crawl-able… >>> More
Understanding Ajax crawling of search site

as seen on Pro Webmasters - Search for 'Pro Webmasters'
I have a couple of questions about Ajax crawling of site, which is kind of search engine itself. The base article explains the mechanism of making AJAX application crawlable. All this stuff with HTML-snapshots is clear and easy to implement, but I cant understand where will Google bot will get "the… >>> More
Need to sanity-check my .htaccess, especially Limit GET POST line for Google repellent

as seen on Pro Webmasters - Search for 'Pro Webmasters'
I need a sanity check on this .htaccess (from a WordPress site) I inherited from a 5 month+ old site. What's the symptom? Google + Bing crawl, but don't index any of the pages. Let me be clear: I'm not mad about "not ranking high." I think something is (accidentally) rejecting search engine indexing… >>> More
Ajax site not being crawled - have escaped fragment, what's wrong? [closed]

as seen on Pro Webmasters - Search for 'Pro Webmasters'
My site is anonkun.com. You can see that it's "ajax" and doesn't load much HTML. Here are some example pages: http://anonkun.com http://anonkun.com/?_escaped_fragment_= http://anonkun.com/stories/Dev-kun---FAQ/6ef881f8-cf48-4f87-a688-c585f23809c5 http://anonkun.com/stories/Dev-kun---FAQ/6ef881f8-cf48-4f87-a688-c585f23809c5… >>> More
How to interpret number of URL errors in Google webmaster tools

as seen on Pro Webmasters - Search for 'Pro Webmasters'
Recently Google has made some changes to Webmaster tools which are explained below: http://googlewebmastercentral.blogspot.com/2012/03/crawl-errors-next-generation.html One thing I could not find out is how to interpret the number of errors over time. At the end of February we've recently migrated… >>> More