How to build a web crawler to find a specific advert, which is in an iframe loaded by Javascript

Posted by ZoFreX on Stack Overflow See other posts from Stack Overflow or by ZoFreX
Published on 2010-02-26T20:37:10Z Indexed on 2010/03/14 2:05 UTC
Read the original article Hit count: 320

Filed under:

web-crawler

|

iframe

|

JavaScript

I'm trying to find all instances of an advert on a website. The advert is in an iframe which is loaded by javascript (it doesn't appear at all if javascript is turned off). Detecting the advert itself is extremely simple, both the name of the flash file and the target of the href always contain a certain string.

What would be the best "starting point" for achieving this? At the moment I'm considering an Adobe AIR app, which could crawl the site and examine the DOM to find the ad, and would run javascript and load the content of the iframe. The other option I can think of is using Firefox as the platform (using maybe GreaseMonkey or Selenium? I don't really know how to leverage Firefox like this).

Does anyone know of anything suitable to build this, or have any suggestions on using Firefox to do it?

Extra details:

Being CPU intensive isn't really an issue, nor is anything depending on a browser being open. This doesn't need to run on a headless server, it will be running on a powerful desktop box. OS is also not an issue. It would be advantageous if the crawler loaded each page multiple times, as the advert is in rotation. While the crawler does need to execute the javascript and load the content of the iframe, it does not need to be able to display flash files.

© Stack Overflow or respective owner

Related posts about web-crawler

web crawler needed

as seen on Stack Overflow - Search for 'Stack Overflow'
does anybody know where i can get a free web crawler that actually works with minimal coding by me. ive googled it and can only find really old ones that dont work or openwebspider which doesnt seem to work. ideally id like to store just the web addresses and which links that page contains any suggestions… >>> More
Building an automatic web crawler

as seen on Stack Overflow - Search for 'Stack Overflow'
I am building a web application crawler that's meant not only to find all the links or pages in a web application, but also perform all the allowed actions in the app (such as pushing buttons, filling forms, notice changes in the DOM even if they did not trigger a request etc.) Basically, this is… >>> More
Appengine Apps Vs Google bot web crawler

as seen on Stack Overflow - Search for 'Stack Overflow'
i built an appengine web app cricket.hover.in. The web app consists of about 15k url's linked in it, But even after a long time of my launch, no pages are indexed on google. Any base link place on my root site hover.in are being indexed with in minutes. but i placed the same link home page of root… >>> More
Extracting data from internet

as seen on Programmers - Search for 'Programmers'
I would like to extract data from internet like www.mozenda.com does but I want to write my own program to do that. Specific data I'm looking for is various event data. Based on my research, I think custom web crawler is my answer but I Would like to confirm the answer and see if there are any suggestion… >>> More
Web crawler update strategy

as seen on Stack Overflow - Search for 'Stack Overflow'
I want to crawl useful resource (like background picture .. ) from certain websites. It is not a hard job, especially with the help of some wonderful projects like scrapy. The problem here is I not only just want crawl this site ONE TIME. I also want to keep my crawl long running and crawl the updated… >>> More

Related posts about iframe

IE6 iframe anchor links moves iframe up

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi, Having a real head scratching moment... I have a site where there is a footer div that always sits at the bottom of the screen (26px high), and above that I have an iFrame which sizes to 100% of the remaining height. This works well. Even clicking on anchor links inside the iframe works as it… >>> More
Scripted iFrame loads contents over current page instead of within an iFrame

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm trying to load a webpage into an iFrame while keeping a content bar on top, but for some pages that I try to load into the iFrame the contents is loaded as if I navigated to that page. I use the following code to dynamically load an iframe: $('#my_iFrame').attr("src","http://www.nytimes.com/2010/03/26/opinion/26ryan… >>> More
Reloading iframe from another iframe

as seen on Stack Overflow - Search for 'Stack Overflow'
Can I reload iframe (say __tree_iframe) from another child iframe (__content_iframe)? Difficulty: IE6/7. UPDATE Obviosly I should use window.opener and find iframe using it iframes property. Apparently, IE doesn't support window.opener. Any ideas about another way? >>> More
IFrame targeting another IFrame

as seen on Stack Overflow - Search for 'Stack Overflow'
I have a parent page containing two iframes. One holds navigation links and the other should display the information pertaining to the navigation link clicked in the first iframe. So I need to target the one iframe from within another iframe.Is this possible? If so how does one go about it? >>> More
Reloading iframe from anpother iframe

as seen on Stack Overflow - Search for 'Stack Overflow'
Can I reload iframe (say __tree_iframe) from another child iframe (__content_iframe)? Difficulty: IE6/7. >>> More