How to mass-fetch pages from a website behind authentication in PHP

Posted by avastreg on Stack Overflow See other posts from Stack Overflow or by avastreg
Published on 2010-05-20T15:33:40Z Indexed on 2010/05/20 15:40 UTC
Read the original article Hit count: 177

Filed under:

php

|

fetch

|

authentication

|

curl

Yes, yes, i know user and password.

I need some trick in php to get logged into a website and retrieve some images/contents, like a normal website.

Obviously with a curl o file_get_contents it doesn't work because i'm not authenticated.

How i can do?

© Stack Overflow or respective owner

Related posts about php

Magento, NGINX, PHP-FPM, APC, MEMCACHED, 16gb Ram CentOS, Spiking PHP-FPM to 100% CPU

as seen on Server Fault - Search for 'Server Fault'
I have been trying to resolve my issue of spiking cpu caused by php-fpm processes. I've reduced the php-fpm config settings to: pm = ondemand pm.max_children = 12 pm.start_servers = 2 pm.min_spare_servers = 2 pm.max_spare_servers = 10 pm.max_requests = 500 php_admin_value[memory_limit] = 128M Problem… >>> More
PHP Pear Installation on CentOS

as seen on Server Fault - Search for 'Server Fault'
[root@ip ~]# yum install php-pear* Reducing CentOS-5 Testing to included packages only Finished Setting up Install Process Package 1:php-pear-1.8.1-2.el5.centos.noarch already installed and latest versio … >>> More
Apache configurations for php "AddType text/html php" or "AddType application/x-httpd-php php .php"

as seen on Server Fault - Search for 'Server Fault'
I am taking over an application server and discover that it contain the following settings: AddType text/html php Although it works, but my understanding is that it should set as following: AddType application/x-httpd-php php .php What are the key differences between the two settings?… >>> More
mod_rewrite settings causes server to throw HTTP 500 errors instead of 404

as seen on Server Fault - Search for 'Server Fault'
Hello. I have a server with VBulletin forum (working under Apache 2.2, CentOS). The default settings for it in .htaccess are as follows: RewriteEngine on RewriteCond %{HTTP_HOST} ^gsmforum\.ru RewriteRule (.*) http://www.gsmforum.ru/$1 [R=301,L] # If you are having problems or are using VirtualDocumentRoot… >>> More
Problems installing Memcache (PECL extension)

as seen on Server Fault - Search for 'Server Fault'
I have installed memcached fine, and now I will need to install PECL extension memcache. Im running RedHat x86_64 es5. The installation gives me this: downloading memcache-2.2.6.tgz ... Starting to download memcache-2.2.6.tgz (35,957 bytes) ..........done: 35,957 bytes 11 source files, building running:… >>> More

Related posts about fetch

'sudo apt-get update' error

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I've been having an issue with 'sudo apt-get update' for several days now. I've tried every proposed solution I could find but to no avail. Here is the output to 'apt-get update'. Ign http://us.archive.ubuntu.com precise InRelease Ign http://us.archive.ubuntu.com precise-updates… >>> More
sudo apt-get update problem

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I have a problem with sudo apt-get update that I can't seem to fix and It's causing problems with alot of installations I want to do. I run Ubuntu 12.04. Ign http://ftp.availo.se precise InRelease Ign http://ftp.availo.se precise-updates InRelease Ign http://ftp.availo… >>> More
Can't upgrade ubuntu 9.xx to 12.04

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I can't upgrade old Ubuntu 9.10 to new, if I check for upgrade it says: Could not download all repository indexes *Failed to fetch ttp://security.ubuntu.com/ubuntu/dists/jaunty-security/main/binary-i386/Packages 404 Not Found Failed to fetch ttp://security.ubuntu.com/ubuntu/dists/jaunty-security/restricted/binary-i386/Packages… >>> More
Cannot update, apt-get cannot fetch index files

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I have a fresh install of Ubuntu 11.10 from the iso 'ubuntu-11.10-desktop-amd64.iso'. I installed this in VMWare Fusion 4.1.1 running on OSX 10.7.3. When setting up the VM, I allowed easy install to take care of creating my user and installing VMWare tools. No problems during installation, everything… >>> More
Fetch all entries exept those with empty column in Rails

as seen on Stack Overflow - Search for 'Stack Overflow'
How do I fetch all 'link' entries exept those with an 'url' column that is blank? My controller looks like this: def show @user = User.find_by_username(params[:username]) @links = @user.links.all respond_to do |format| format.html # show.html.erb format.xml { render :xml… >>> More