How to extract content from other websites automatically?

Posted by Vijay on Stack Overflow See other posts from Stack Overflow or by Vijay
Published on 2010-02-15T11:10:30Z Indexed on 2010/03/18 7:11 UTC
Read the original article Hit count: 333

Filed under:

php

|

automation

I want to extract a specific data from the website from its pages...

I dont want to get all the contents of a specific page but i need only some portion (may be data only inside a table or content_div) and i want to do it repeatedly along all the pages of the website..

How can i do that?

© Stack Overflow or respective owner

Related posts about php

Magento, NGINX, PHP-FPM, APC, MEMCACHED, 16gb Ram CentOS, Spiking PHP-FPM to 100% CPU

as seen on Server Fault - Search for 'Server Fault'
I have been trying to resolve my issue of spiking cpu caused by php-fpm processes. I've reduced the php-fpm config settings to: pm = ondemand pm.max_children = 12 pm.start_servers = 2 pm.min_spare_servers = 2 pm.max_spare_servers = 10 pm.max_requests = 500 php_admin_value[memory_limit] = 128M Problem… >>> More
PHP Pear Installation on CentOS

as seen on Server Fault - Search for 'Server Fault'
[root@ip ~]# yum install php-pear* Reducing CentOS-5 Testing to included packages only Finished Setting up Install Process Package 1:php-pear-1.8.1-2.el5.centos.noarch already installed and latest versio … >>> More
Apache configurations for php "AddType text/html php" or "AddType application/x-httpd-php php .php"

as seen on Server Fault - Search for 'Server Fault'
I am taking over an application server and discover that it contain the following settings: AddType text/html php Although it works, but my understanding is that it should set as following: AddType application/x-httpd-php php .php What are the key differences between the two settings?… >>> More
mod_rewrite settings causes server to throw HTTP 500 errors instead of 404

as seen on Server Fault - Search for 'Server Fault'
Hello. I have a server with VBulletin forum (working under Apache 2.2, CentOS). The default settings for it in .htaccess are as follows: RewriteEngine on RewriteCond %{HTTP_HOST} ^gsmforum\.ru RewriteRule (.*) http://www.gsmforum.ru/$1 [R=301,L] # If you are having problems or are using VirtualDocumentRoot… >>> More
Problems installing Memcache (PECL extension)

as seen on Server Fault - Search for 'Server Fault'
I have installed memcached fine, and now I will need to install PECL extension memcache. Im running RedHat x86_64 es5. The installation gives me this: downloading memcache-2.2.6.tgz ... Starting to download memcache-2.2.6.tgz (35,957 bytes) ..........done: 35,957 bytes 11 source files, building running:… >>> More

Related posts about automation

Documenting applications - automation / semi-automation for screenshots?

as seen on Stack Overflow - Search for 'Stack Overflow'
For me one of the biggest bores of being a developer is writing user documentation. (I am referring to the stuff that gets exported into PDF files files that ship with the product, not comments in code here). The task off adding or updating new bits of text to the existing documentation is OK. However… >>> More
Why the cucumber features keeps running even though it fails?

as seen on Stack Overflow - Search for 'Stack Overflow'
Its a rails 2.3.5 app. I'm using the rspec and cucumber for testing. When I run autospec, it runs correctly with the warning (Not running features. To run features in autotest, set AUTOFEATURE=true.) as below: [~/rails_apps/automation (campaign)?] ? autospec (Not running features. To run features… >>> More
Automation testing tool for Regression testing of desktop application

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi I am working on a desktop application which uses Infragistic grids. We need to automate the regression tests for same. QTP alone does not support this, we need to buy new plug in for same which my company is not very much interested in. Do we have any open source tool for automating regression… >>> More
automation of data format conversion to parent child format

as seen on Stack Overflow - Search for 'Stack Overflow'
This is an excel sheet which has only a single column filled for each row. (explanation : all CITY categories fall under V21 , all handset categories fall under CityJ and so on ) V21 CITYR CITYJ HandsetS HandsetHW HandsetHA … >>> More
Automation: how to automate transforming .doc to .docx?

as seen on Stack Overflow - Search for 'Stack Overflow'
I have a bunch of .doc files in a folder which I need to convert to .docx. To manually convert the .doc to .docx is pretty simple: Open .doc in Word 2007 Click on Save As... Save it as .docx However, doing this for hundreds of files definitely ain't fun. =p How would you automate this? >>> More