How can I get the ultimate URL without fetching the pages using perl and LWP?

Posted by planetp on Stack Overflow See other posts from Stack Overflow or by planetp
Published on 2010-03-18T13:20:03Z Indexed on 2010/03/18 13:21 UTC
Read the original article Hit count: 424

Filed under:

perl

|

lwp

|

redirect

I'm doing some web scraping using perl's LWP. I need to process a set of URLs, some of which may redirect (1 or more times).

How can I get ultimate url with all redirects resolved, using HEAD method ?

© Stack Overflow or respective owner

Related posts about perl

Munin on Centos 6 - missing perl MODULE_COMPAT_5.8.8

as seen on Server Fault - Search for 'Server Fault'
I'm trying to install Munin on a new VPS through yum install munin but I keep getting an error about a missing perl module: Requires: perl(:MODULE_COMPAT_5.8.8). This is the perl version currently installed: v5.10.1. I've searched all around and still haven't found a solution for this. Here's the… >>> More
Pain removing a perl rootkit

as seen on Server Fault - Search for 'Server Fault'
So, we host a geoservice webserver thing at the office. Someone apparently broke into this box (probably via ftp or ssh), and put some kind of irc-managed rootkit thing. Now I'm trying to clean the whole thing up, I found the process pid who tries to connect via irc, but i can't figure out who's… >>> More
How To Avoid a Perl script calling an Another Perl Script

as seen on Stack Overflow - Search for 'Stack Overflow'
Hello, i am calling a perl script client.pl from a main script to capture the output of client.pl in @output. is there anyway to avoid the use of these two files so i can use the output of client.pl in main.pl itself here is my code.... main.pl ======= my @output = readpipe("client.pl"); client… >>> More
Perl :how to sort dates in perl

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi, How can I sort the dates in perl. my @dates = ( "02/11/2009" , "12/20/2001" , "11/21/2010" ) ; I have above dates in my array . How can I sort those dates... ? My date format is dd/mm/YYYY. >>> More
please suggest a perl book exclusively for perl programs

as seen on Stack Overflow - Search for 'Stack Overflow'
I want tha name of a perl book for only PERL PROGRAMS. The reason behind is I want to improve my programming skill in perl >>> More

Related posts about lwp

Making REST request using LWP::Simple

as seen on Stack Overflow - Search for 'Stack Overflow'
I am trying to use LWP::Simple to make a GET request to a REST service. Here's the simple code: use LWP::Simple; $uri = "http://api.stackoverflow.com/0.8/questions/tagged/php"; $jsonresponse= get $uri; print $jsonresponse; On my local machine, running Ubuntu 10.4, and Perl version 5.10.1: farhan@farhan-lnx:~$… >>> More
Perl LWP::UserAgent mishandling UTF-8 response

as seen on Stack Overflow - Search for 'Stack Overflow'
When I use LWP::UserAgent to retrieve content encoded in UTF-8 it seems LWP::UserAgent doesn't handle the encoding correctly. Here's the output after setting the Command Prompt window to Unicode by the command chcp 65001 Note that this initially gives the appearance that all is well, but I think… >>> More
How can I get the ultimate URL without fetching the pages using perl and LWP?

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm doing some web scraping using perl's LWP. I need to process a set of URLs, some of which may redirect (1 or more times). How can I get ultimate url with all redirects resolved, using HEAD method ? >>> More
How do I use Perl's LWP to log in to a web application?

as seen on Stack Overflow - Search for 'Stack Overflow'
I would like to write a script to login to a web application and then move to other parts of the application: use HTTP::Request::Common qw(POST); use LWP::UserAgent; use Data::Dumper; $ua = LWP::UserAgent->new(keep_alive=>1); my $req = POST "http://example.com:5002/index.php", [ user_name… >>> More
lwp-rget changes file format automatically to .bin?

as seen on Super User - Search for 'Super User'
Im trying to recursively get an html page to get everything that's there and save it to my directory. Lwp-rget from perl allows me to do this but I'm having the problem that its getting all the files and changing them from .rpm, .xml, .html, etc to .bin. I've been trying to maybe use the --keepext=application/xml… >>> More