Perl - WWW::Mechanize Cookie Session Id is being reset with every get(), how to make it stop?
Posted
by Phill Pafford
on Stack Overflow
See other posts from Stack Overflow
or by Phill Pafford
Published on 2010-04-27T14:53:02Z
Indexed on
2010/04/27
15:53 UTC
Read the original article
Hit count: 584
So I'm scraping a site that I have access to via HTTPS, I can login and start the process but each time I hit a new page (URL) the cookie Session Id changes. How do I keep the logged in Cookie Session Id?
#!/usr/bin/perl -w
use strict;
use warnings;
use WWW::Mechanize;
use HTTP::Cookies;
use LWP::Debug qw(+);
use HTTP::Request;
use LWP::UserAgent;
use HTTP::Request::Common;
my $un = 'username';
my $pw = 'password';
my $url = 'https://subdomain.url.com/index.do';
my $agent = WWW::Mechanize->new(cookie_jar => {}, autocheck => 0);
$agent->{onerror}=\&WWW::Mechanize::_warn;
$agent->agent('Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.3) Gecko/20100407 Ubuntu/9.10 (karmic) Firefox/3.6.3');
$agent->get($url);
$agent->form_name('form');
$agent->field(username => $un);
$agent->field(password => $pw);
$agent->click("Log In");
print "After Login Cookie: ";
print $agent->cookie_jar->as_string();
print "\n\n";
my $searchURL='https://subdomain.url.com/search.do';
$agent->get($searchURL);
print "After Search Cookie: ";
print $agent->cookie_jar->as_string();
print "\n";
The output:
After Login Cookie: Set-Cookie3: JSESSIONID=367C6D; path="/thepath"; domain=subdomina.url.com; path_spec; secure; discard; version=0
After Search Cookie: Set-Cookie3: JSESSIONID=855402; path="/thepath"; domain=subdomain.com.com; path_spec; secure; discard; version=0
Also I think the site requires a CERT (Well in the browser it does), would this be the correct way to add it?
$ENV{HTTPS_CERT_FILE} = 'SUBDOMAIN.URL.COM'; ## Insert this after the use HTTP::Request...
Also for the CERT In using the first option in this list, is this correct?
X.509 Certificate (PEM)
X.509 Certificate with chain (PEM)
X.509 Certificate (DER)
X.509 Certificate (PKCS#7)
X.509 Certificate with chain (PKCS#7)
© Stack Overflow or respective owner