How to know if the website being scraped has changed?

Posted by Lost_in_code on Stack Overflow See other posts from Stack Overflow or by Lost_in_code
Published on 2010-03-27T17:52:13Z Indexed on 2010/03/27 17:53 UTC
Read the original article Hit count: 163

Filed under:

scraping

|

webscraping

|

php

I'm using PHP to scrape a website and collect some data. It's all done without using regex. I'm using php's explode() method to find particular HTML tags instead.

It is possible that if the structure of the website changes (CSS, HTML), then wrong data may be collected by the scraper. So the question is - how do I know if the HTML structure has changed? How to identify this before storing any data to my database to avoid wrong data being stored.

© Stack Overflow or respective owner

Related posts about scraping

Screen-scraping of a secure page of any site on https:// with asp.net in C#

as seen on Stack Overflow - Search for 'Stack Overflow'
I've done site scraping of secure page of any site on http:// but when I am trying to scrap any site on https:// then i always scrape the login page not secure page. Please advice what should i do for scraping a secure page of any site on https://. >>> More
looking for alternative to Webzinc .NET , screen scraping, web automation library for .net

as seen on Stack Overflow - Search for 'Stack Overflow'
i came across this .net library http://www.webzinc.com/online/faq.aspx however, i was wondering if there was a free alternative out there ? >>> More
PHP Screen Scraping Class

as seen on Bradino - Search for 'Bradino'
After some positive feedback I have decided to continue to develop the PHP Screen Scraping class. This post will server as the permanent home for the class. Download PHP Screen Scraping Class Updates 20009-07-30 Added setHeader() function >>> More
Alert Log Scraping with Oracle&#146;s ADRCI Utility

as seen on Internet.com - Search for 'Internet.com'
Oracles new ADR with command interface shows promise for database administrators who like to script their own solution for quickly scraping the alert log and automatically looking for errors. >>> More
Web scraping etiquette

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm considering writing a simple web scraping application to extract information from a website that does not seem to specifically prohibit this. I've checked for other alternatives (eg RSS, web service) to get this information, but there are none available at this stage. Despite this I've also… >>> More

Related posts about webscraping

Webscraping Google tasks via Google Calendar

as seen on Stack Overflow - Search for 'Stack Overflow'
As gmail and the task api is not available everywhere (eg: some companies block gmail but not calendar), is there a way to scrap google task through the calendar web interface ? The solution can be to use jQuery/Jaxer or a pointer to a browser script/plugin. >>> More
R webscraping: interrogating for date and importance

as seen on Stack Overflow - Search for 'Stack Overflow'
I am able to webscrape a table from a webpage containing news library(XML) webpage <- "http://www.tradingeconomics.com/calendar" tables <- readHTMLTable(webpage ) n.rows <- unlist(lapply(tables, function(t) dim(t)[1])) dfcal <- as.data.frame(tables$calendar) However I do not know… >>> More
How can I use R (Rcurl/XML packages ?!) to scrap this webpage ?

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi all, I have a (somewhat complex) webscraping challenge that I wish to accomplish and would love for some direction (to whatever level you feel like sharing) here goes: I would like to go through all the "species pages" present in this link: http://gtrnadb.ucsc.edu/ So for each of them I will… >>> More
HTML Agility Pack Screen Scraping XPATH isn't returning data

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm attempting to write a screen scraper for Digikey that will allow our company to keep accurate track of pricing, part availability and product replacements when a part is discontinued. There seems to be a discrepancy between the XPATH that I'm seeing in Chrome Devtools as well as Firebug on Firefox… >>> More
Is there a jQuery webscraper out there?

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm trying to pullout some info from an external site using jQuery and Adobe AIR. Right now I'm using a hidden div and jQuery's load function to load fragments of the external site, once the info is loaded I parse some info with selectors. This is fine but it's kinda dirty and I need to perform this… >>> More