Search Results

Search found 19 results on 1 pages for 'cookielib'.

Page 1/1 | 1 

  • Python - urllib2 & cookielib

    - by Adrian
    I am trying to open the following website and retrieve the initial cookie and use it for the second url-open BUT if you run the following code it outputs 2 different cookies. How do I use the initial cookie for the second url-open? import cookielib, urllib2 cj = cookielib.CookieJar() opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj)) home = opener.open('https://www.idcourts.us/repository/start.do') print cj search = opener.open('https://www.idcourts.us/repository/partySearch.do') print cj Output shows 2 different cookies every time as you can see: <cookielib.CookieJar[<Cookie JSESSIONID=0DEEE8331DE7D0DFDC22E860E065085F for www.idcourts.us/repository>]> <cookielib.CookieJar[<Cookie JSESSIONID=E01C2BE8323632A32DA467F8A9B22A51 for www.idcourts.us/repository>]>

    Read the article

  • Storing cookielib cookies in a database

    - by Mridang Agarwalla
    Hi, I'm using the cookielib module to handle HTTP cookies when using the urllib2 module in Python 2.6 in a way similar to this snippet: import cookielib, urllib2 cj = cookielib.CookieJar() opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj)) r = opener.open("http://example.com/") I'd like to store the cookies in a database. I don't know whats better - serialize the CookieJar object and store it or extract the cookies from the CookieJar and store that. I don't know which one's better or how to implement either of them. I should be also be able to recreate the CookieJar object. Could someone help me out with the above? Thanks in advance.

    Read the article

  • Scrapy issue with iTunes' AppStore

    - by Eric
    I am using Scrapy to fetch some data from iTunes' AppStore database. I start with this list of apps: http://itunes.apple.com/us/genre/mobile-software-applications/id36?mt=8 In the following code I have used the simplest regex which targets all apps in the US store. from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor from scrapy.contrib.spiders import CrawlSpider, Rule class AppStoreSpider(CrawlSpider): domain_name = 'itunes.apple.com' start_urls = ['http://itunes.apple.com/us/genre/mobile-software-applications/id6015?mt=8'] rules = ( Rule(SgmlLinkExtractor(allow='itunes\.apple\.com/us/app'), 'parse_app', follow=True, ), ) def parse_app(self, response): .... SPIDER = AppStoreSpider() When I run it I receive the following: [itunes.apple.com] DEBUG: Crawled (200) <GET http://itunes.apple.com/us/genre/mobile-software-applications/id6015?mt=8> (referer: None) [itunes.apple.com] DEBUG: Filtered offsite request to 'itunes.apple.com': <GET http://itunes.apple.com/us/app/bloomberg/id281941097?mt=8> As you can see, when it starts crawling the first page it says: "Filtered offsite request to 'itunes.apple.com'". and then the spider stops.. it also returns this message: [ScrapyHTTPPageGetter,client] /usr/lib/python2.5/cookielib.py:1577: exceptions.UserWarning: cookielib bug! Traceback (most recent call last): File "/usr/lib/python2.5/cookielib.py", line 1575, in make_cookies parse_ns_headers(ns_hdrs), request) File "/usr/lib/python2.5/cookielib.py", line 1532, in _cookies_from_attrs_set cookie = self._cookie_from_cookie_tuple(tup, request) File "/usr/lib/python2.5/cookielib.py", line 1451, in _cookie_from_cookie_tuple if version is not None: version = int(version) ValueError: invalid literal for int() with base 10: '"1"' I have used the same script for other website and I didn't have this problem. Any suggestion?

    Read the article

  • Python form POST using urllib2 (also question on saving/using cookies)

    - by morpheous
    I am trying to write a function to post form data and save returned cookie info in a file so that the next time the page is visited, the cookie information is sent to the server (i.e. normal browser behavior). I wrote this relatively easily in C++ using curlib, but have spent almost an entire day trying to write this in Python, using urllib2 - and still no success. This is what I have so far: import urllib, urllib2 import logging # the path and filename to save your cookies in COOKIEFILE = 'cookies.lwp' cj = None ClientCookie = None cookielib = None logger = logging.getLogger(__name__) # Let's see if cookielib is available try: import cookielib except ImportError: logger.debug('importing cookielib failed. Trying ClientCookie') try: import ClientCookie except ImportError: logger.debug('ClientCookie isn\'t available either') urlopen = urllib2.urlopen Request = urllib2.Request else: logger.debug('imported ClientCookie succesfully') urlopen = ClientCookie.urlopen Request = ClientCookie.Request cj = ClientCookie.LWPCookieJar() else: logger.debug('Successfully imported cookielib') urlopen = urllib2.urlopen Request = urllib2.Request # This is a subclass of FileCookieJar # that has useful load and save methods cj = cookielib.LWPCookieJar() login_params = {'name': 'anon', 'password': 'pass' } def login(theurl, login_params): init_cookies(); data = urllib.urlencode(login_params) txheaders = {'User-agent' : 'Mozilla/4.0 (compatible; MSIE 5.5; Windows NT)'} try: # create a request object req = Request(theurl, data, txheaders) # and open it to return a handle on the url handle = urlopen(req) except IOError, e: log.debug('Failed to open "%s".' % theurl) if hasattr(e, 'code'): log.debug('Failed with error code - %s.' % e.code) elif hasattr(e, 'reason'): log.debug("The error object has the following 'reason' attribute :"+e.reason) sys.exit() else: if cj is None: log.debug('We don\'t have a cookie library available - sorry.') else: print 'These are the cookies we have received so far :' for index, cookie in enumerate(cj): print index, ' : ', cookie # save the cookies again cj.save(COOKIEFILE) #return the data return handle.read() # FIXME: I need to fix this so that it takes into account any cookie data we may have stored def get_page(*args, **query): if len(args) != 1: raise ValueError( "post_page() takes exactly 1 argument (%d given)" % len(args) ) url = args[0] query = urllib.urlencode(list(query.iteritems())) if not url.endswith('/') and query: url += '/' if query: url += "?" + query resource = urllib.urlopen(url) logger.debug('GET url "%s" => "%s", code %d' % (url, resource.url, resource.code)) return resource.read() When I attempt to log in, I pass the correct username and pwd,. yet the login fails, and no cookie data is saved. My two questions are: can anyone see whats wrong with the login() function, and how may I fix it? how may I modify the get_page() function to make use of any cookie info I have saved ?

    Read the article

  • [Python] Different work of the script in Windows and in FreeBSD

    - by www.yegorov-p.ru
    Hello. I'm writing some script, that works with web-servers. So, I have the following code: client = suds.client.Client(WSDLfile) client.service.Login('mylogin', 'mypass') print client.options.transport.cookiejar ####### sessnum = str(client.options.transport.cookiejar).split(' ')[1] client = suds.client.Client( WSDLfile, headers= { 'Set-Cookie' : sessnum } ) When running in FreeBSD, it returns <cookielib.CookieJar[<Cookie sessnum=9WAXQ25D37XY535F6SZ3GXKSCTZG8CVJ for .IP.IP.IP.IP/>]> but in Windows it returns <cookielib.CookieJar[]> How can I fix it?

    Read the article

  • python coockie,request another page

    - by polovinamozga
    #!/usr/bin/python # -*- coding: utf-8 -*- import urllib2 import urllib import httplib import Cookie import cookielib Login = 'user' Password = 'password' Domain = 'inbox.ru' Auth = 'https://auth.mail.ru/cgi-bin/auth' cj = cookielib.CookieJar() opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj)) login_data = urllib.urlencode({'Login' : Login, 'Domain' :Domain, 'Password' : Password }) opener.open('https://auth.mail.ru/cgi-bin/auth', login_data) resp = opener.open('https://auth.mail.ru/cgi-bin/auth').read() print resp.decode('cp1251') #output page in cp1251 When script sucessfully executed i see in print resp.decode('cp1251') my page with auth. But when a try to request another page for example http://my.mail.ru i see autorization request. How i can use cookie with another page?

    Read the article

  • Python urllib3 and how to handle cookie support?

    - by bigredbob
    So I'm looking into urllib3 because it has connection pooling and is thread safe (so performance is better, especially for crawling), but the documentation is... minimal to say the least. urllib2 has build_opener so something like: #!/usr/bin/python import cookielib, urllib2 cj = cookielib.CookieJar() opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj)) r = opener.open("http://example.com/") But urllib3 has no build_opener method, so the only way I have figured out so far is to manually put it in the header: #!/usr/bin/python import urllib3 http_pool = urllib3.connection_from_url("http://example.com") myheaders = {'Cookie':'some cookie data'} r = http_pool.get_url("http://example.org/", headers=myheaders) But I am hoping there is a better way and that one of you can tell me what it is. Also can someone tag this with "urllib3" please.

    Read the article

  • How to login with python and save cookie, then use that info to view page for member only

    - by patcsong
    I am quite new in python and trying to write a script to login to the page at http://ryushare.com/login.python. I have try many attempt, but it fails to login and i have no idea why. After login to the page, I wish to get the return of http://ryushare.com/file-manager.python Here's the code I try to attempt by reading the example from others. import urllib, urllib2, cookielib username = 'myusername' password = 'mypassword' cj = cookielib.CookieJar() opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj)) login_data = urllib.urlencode({'login' : username, 'password' : password}) opener.open('http://www.ryushare.com/login.python', login_data) resp = opener.open('http://ryushare.com/file-manager.python') print resp.read() I check the source code of the login page, it said the username and password value is "login and password" so i change it. I have try some other example which can be found here like google news feed, It also can not able to login : (

    Read the article

  • python mechanize.browser submit() related problem

    - by paul
    Hello All im making some script with mechanize.browser module. one of problem is all other thing is ok, but when submit() form,it not working, so i was found some suspicion source part. in the html source i was found such like following. <form method="post" onsubmit="return loginCheck(this)" name="FRMLOGIN"/> im thinking, loginCheck(this) making problem when submit form. but how to handle this kind of javascript function with mechanize module ,so i can successfully submit form and can receive result? folloing is my current script source. if anyone can help me ..much appreciate!! # -*- coding: cp949-*- import sys,os import mechanize, urllib import cookielib from BeautifulSoup import BeautifulSoup,BeautifulStoneSoup,Tag import datetime, time, socket import re,sys,os,mechanize,urllib,time br = mechanize.Browser() cj = cookielib.LWPCookieJar() br.set_cookiejar(cj) # Browser options br.set_handle_equiv(True) br.set_handle_gzip(True) br.set_handle_redirect(True) br.set_handle_referer(True) br.set_handle_robots(False) # Follows refresh 0 but not hangs on refresh > 0 br.set_handle_refresh(mechanize._http.HTTPRefreshProcessor(), max_time=1) # Want debugging messages? br.set_debug_http(True) br.set_debug_redirects(True) br.set_debug_responses(True) # User-Agent (this is cheating, ok?) br.addheaders = [('User-agent', 'Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.0.6')] br.open('http://user.buddybuddy.co.kr/Login/LoginForm.asp?URL=') html = br.response().read() print html br.select_form(name='FRMLOGIN') print br.viewing_html() br.form['ID']='zero1zero2' br.form['PWD']='012045' br.submit() print br.response().read()

    Read the article

  • Bitbucket API authentication with Python's HTTPBasicAuthHandler

    - by jbochi
    I'm trying to get the list of issues on a private repository using bitbucket's API. I have confirmed that HTTP Basic authentication works with hurl, but I am unable to authenticate in Python. Adapting the code from this tutorial, I have written the following script. import cookielib import urllib2 class API(): api_url = 'http://api.bitbucket.org/1.0/' def __init__(self, username, password): self._opener = self._create_opener(username, password) def _create_opener(self, username, password): cj = cookielib.LWPCookieJar() cookie_handler = urllib2.HTTPCookieProcessor(cj) password_manager = urllib2.HTTPPasswordMgrWithDefaultRealm() password_manager.add_password(None, self.api_url, username, password) auth_handler = urllib2.HTTPBasicAuthHandler(password_manager) opener = urllib2.build_opener(cookie_handler, auth_handler) return opener def get_issues(self, username, repository): query_url = self.api_url + 'repositories/%s/%s/issues/' % (username, repository) try: handler = self._opener.open(query_url) except urllib2.HTTPError, e: print e.headers raise e return handler.read() api = API(username='my_username', password='XXXXXXXX') api.get_issues('my_username', 'my_repository') results in: >>> Server: nginx/0.7.62 Date: Mon, 19 Apr 2010 16:15:06 GMT Content-Type: text/plain Connection: close Vary: Authorization,Cookie Content-Length: 9 Traceback (most recent call last): File "C:/USERS/personal/bitbucket-burndown/bitbucket-api.py", line 29, in <module> print api.get_issues('my_username', 'my_repository') File "C:/USERS/personal/bitbucket-burndown/bitbucket-api.py", line 25, in get_issues raise e HTTPError: HTTP Error 401: UNAUTHORIZED api.get_issues('jespern', 'bitbucket') works like a charm. What's wrong with my code?

    Read the article

  • Using CookieJar in Python to log in to a website from "Google App Engine". What's wrong here?

    - by brilliant
    Hello everybody, I've been trying to find a python code that would log in to my mail box on yahoo.com from "Google App Engine" . Here (click here to see that page) I was given this code: import urllib, urllib2, cookielib url = "https://login.yahoo.com/config/login?" form_data = {'login' : 'my-login-here', 'passwd' : 'my-password-here'} jar = cookielib.CookieJar() opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(jar)) form_data = urllib.urlencode(form_data) # data returned from this pages contains redirection resp = opener.open(url, form_data) # yahoo redirects to http://my.yahoo.com, so lets go there instead resp = opener.open('http://mail.yahoo.com') print resp.read() The author of this script looked into HTML script of yahoo log-in form and came up with this script. That log-in form contains two fields, one for users' Yahoo! ID and another one is for users' password. Here is how HTML code of that page for both of those fields looks like: User ID field: <input type="text" maxlength="96" class="yreg_ipt" size="17" value="" id="username" name="login"> Password field: <input type="password" maxlength="64" class="yreg_ipt" size="17" value="" id="passwd" name="passwd"> However, when I uploaded this code to Google App Engine I discovered that this log-in form keeps coming back to me, which, I assume, means that logging-in process didn't succeed. Why is it so?

    Read the article

  • Using CookieJar in Python to log in to a website from "Google App Engine". What's wrong here?

    - by brilliant
    Hello everybody, I've been trying to find a python code that would log in to my mail box on yahoo.com from "Google App Engine" . Here (click here to see that page) I was given this code: import urllib, urllib2, cookielib url = "https://login.yahoo.com/config/login?" form_data = {'login' : 'my-login-here', 'passwd' : 'my-password-here'} jar = cookielib.CookieJar() opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(jar)) form_data = urllib.urlencode(form_data) # data returned from this pages contains redirection resp = opener.open(url, form_data) # yahoo redirects to http://my.yahoo.com, so lets go there instead resp = opener.open('http://mail.yahoo.com') print resp.read() The author of this script looked into HTML script of yahoo log-in form and came up with this script. That log-in form contains two fields, one for users' Yahoo! ID and another one is for users' password. Here is how HTML code of that page for both of those fields looks like: User ID field: <input type="text" maxlength="96" class="yreg_ipt" size="17" value="" id="username" name="login"> Password field: <input type="password" maxlength="64" class="yreg_ipt" size="17" value="" id="passwd" name="passwd"> However, when I uploaded this code to Google App Engine I discovered that this log-in form keeps coming back to me, which, I assume, means that logging-in process didn't succeed. Why is it so?

    Read the article

  • Cross-site request forgery protections: Where do I put all these lines?

    - by brilliant
    Hello, I was looking for a python code that would be able to log in from "Google App Engine" to some of my accounts on some websites (like yahoo or eBay) and was given this code: import urllib, urllib2, cookielib url = "https://login.yahoo.com/config/login?" form_data = {'login' : 'my-login-here', 'passwd' : 'my-password-here'} jar = cookielib.CookieJar() opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(jar)) form_data = urllib.urlencode(form_data) # data returned from this pages contains redirection resp = opener.open(url, form_data) # yahoo redirects to http://my.yahoo.com, so lets go there instead resp = opener.open('http://mail.yahoo.com') print resp.read() Unfortunately, this code didn't work, so I asked another question here and one supporter among other things said this: "You send MD5 hash and not plain password. Also you'd have to play along with all kinds of CSRF protections etc. that they're implementing. Look: <input type="hidden" name=".tries" value="1"> <input type="hidden" name=".src" value="ym"> <input type="hidden" name=".md5" value=""> <input type="hidden" name=".hash" value=""> <input type="hidden" name=".js" value=""> <input type="hidden" name=".last" value=""> <input type="hidden" name="promo" value=""> <input type="hidden" name=".intl" value="us"> <input type="hidden" name=".bypass" value=""> <input type="hidden" name=".partner" value=""> <input type="hidden" name=".u" value="bd5tdpd5rf2pg"> <input type="hidden" name=".v" value="0"> <input type="hidden" name=".challenge" value="5qUiIPGVFzRZ2BHhvtdGXoehfiOj"> <input type="hidden" name=".yplus" value=""> <input type="hidden" name=".emailCode" value=""> <input type="hidden" name="pkg" value=""> <input type="hidden" name="stepid" value=""> <input type="hidden" name=".ev" value=""> <input type="hidden" name="hasMsgr" value="0"> <input type="hidden" name=".chkP" value="Y"> <input type="hidden" name=".done" value="http://mail.yahoo.com"> <input type="hidden" name=".pd" value="ym_ver=0&c=&ivt=&sg="> I am not quite sure where he got all these lines from and where in my code I am supposed to add them. Do You have any idea? I know I was supposed to ask him this question first, and I did, but he never returned, so I decided to ask a separate question here.

    Read the article

  • How to save POST&GET headers of a web page with "Wireshark"?

    - by brilliant
    Hello everybody, I've been trying to find a python code that would log in to my mail box on yahoo.com from "Google App Engine". I was given this code: import urllib, urllib2, cookielib url = "https://login.yahoo.com/config/login?" form_data = {'login' : 'my-login-here', 'passwd' : 'my-password-here'} jar = cookielib.CookieJar() opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(jar)) form_data = urllib.urlencode(form_data) # data returned from this pages contains redirection resp = opener.open(url, form_data) # yahoo redirects to http://my.yahoo.com, so lets go there instead resp = opener.open('http://mail.yahoo.com') print resp.read() The author of this script looked into HTML script of yahoo log-in form and came up with this script. That log-in form contains two fields, one for users' Yahoo! ID and another one is for users' password. However, when I tried this code out (substituting mu real Yahoo login for 'my-login-here' and my real password for 'my-password-here'), it just return the log-in form back to me, which means that something didn't work right. Another supporter suggested that I should send an MD5 hash of my password, rather than a plain password. He also noted that in that log-in form there are a lot other hidden fields besides login and password fields (he called them "CSRF protections") that I would also have to deal with: <input type="hidden" name=".tries" value="1"> <input type="hidden" name=".src" value="ym"> <input type="hidden" name=".md5" value=""> <input type="hidden" name=".hash" value=""> <input type="hidden" name=".js" value=""> <input type="hidden" name=".last" value=""> <input type="hidden" name="promo" value=""> <input type="hidden" name=".intl" value="us"> <input type="hidden" name=".bypass" value=""> <input type="hidden" name=".partner" value=""> <input type="hidden" name=".u" value="bd5tdpd5rf2pg"> <input type="hidden" name=".v" value="0"> <input type="hidden" name=".challenge" value="5qUiIPGVFzRZ2BHhvtdGXoehfiOj"> <input type="hidden" name=".yplus" value=""> <input type="hidden" name=".emailCode" value=""> <input type="hidden" name="pkg" value=""> <input type="hidden" name="stepid" value=""> <input type="hidden" name=".ev" value=""> <input type="hidden" name="hasMsgr" value="0"> <input type="hidden" name=".chkP" value="Y"> <input type="hidden" name=".done" value="http://mail.yahoo.com"> He said that I should do the following: Simulate normal login and save login page that I get; Save POST&GET headers with "Wireshark"; Compare login page with those headers and see what fields I need to include with my request; I really don't know how to carry out the first two of these three steps. I have just downloaded "Wireshark" and have tried capturing some packets there. However, I don't know how to "simulate normal login and save the login page". Also, I don't how to save POST$GET headers with "Wireshark". Can anyone, please, guide me through these two steps in "Wireshark"? Or at least tell me what I should start with. Thank You.

    Read the article

  • Python and mechanize login script

    - by Perun
    Hi fellow programmers! I am trying to write a script to login into my universities "food balance" page using python and the mechanize module... This is the page I am trying to log into: http://www.wcu.edu/11407.asp The website has the following form to login: <FORM method=post action=https://itapp.wcu.edu/BanAuthRedirector/Default.aspx><INPUT value=https://cf.wcu.edu/busafrs/catcard/idsearch.cfm type=hidden name=wcuirs_uri> <P><B>WCU ID Number<BR></B><INPUT maxLength=12 size=12 type=password name=id> </P> <P><B>PIN<BR></B><INPUT maxLength=20 type=password name=PIN> </P> <P></P> <P><INPUT value="Request Access" type=submit name=submit> </P></FORM> From this we know that I need to fill in the following fields: 1. name=id 2. name=PIN With the action: action=https://itapp.wcu.edu/BanAuthRedirector/Default.aspx This is the script I have written thus far: #!/usr/bin/python2 -W ignore import mechanize, cookielib from time import sleep url = 'http://www.wcu.edu/11407.asp' myId = '11111111111' myPin = '22222222222' # Browser #br = mechanize.Browser() #br = mechanize.Browser(factory=mechanize.DefaultFactory(i_want_broken_xhtml_support=True)) br = mechanize.Browser(factory=mechanize.RobustFactory()) # Use this because of bad html tags in the html... # Cookie Jar cj = cookielib.LWPCookieJar() br.set_cookiejar(cj) # Browser options br.set_handle_equiv(True) br.set_handle_gzip(True) br.set_handle_redirect(True) br.set_handle_referer(True) br.set_handle_robots(False) # Follows refresh 0 but not hangs on refresh > 0 br.set_handle_refresh(mechanize._http.HTTPRefreshProcessor(), max_time=1) # User-Agent (fake agent to google-chrome linux x86_64) br.addheaders = [('User-agent','Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.56 Safari/535.11'), ('Accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'), ('Accept-Encoding', 'gzip,deflate,sdch'), ('Accept-Language', 'en-US,en;q=0.8'), ('Accept-Charset', 'ISO-8859-1,utf-8;q=0.7,*;q=0.3')] # The site we will navigate into br.open(url) # Go though all the forms (for debugging only) for f in br.forms(): print f # Select the first (index two) form br.select_form(nr=2) # User credentials br.form['id'] = myId br.form['PIN'] = myPin br.form.action = 'https://itapp.wcu.edu/BanAuthRedirector/Default.aspx' # Login br.submit() # Wait 10 seconds sleep(10) # Save to a file f = file('mycatpage.html', 'w') f.write(br.response().read()) f.close() Now the problem... For some odd reason the page I get back (in mycatpage.html) is the login page and not the expected page that displays my "cat cash balance" and "number of block meals" left... Does anyone have any idea why? Keep in mind that everything is correct with the header files and while the id and pass are not really 111111111 and 222222222, the correct values do work with the website (using a browser...) Thanks in advance EDIT Another script I tried: from urllib import urlopen, urlencode import urllib2 import httplib url = 'https://itapp.wcu.edu/BanAuthRedirector/Default.aspx' myId = 'xxxxxxxx' myPin = 'xxxxxxxx' data = { 'id':myId, 'PIN':myPin, 'submit':'Request Access', 'wcuirs_uri':'https://cf.wcu.edu/busafrs/catcard/idsearch.cfm' } opener = urllib2.build_opener() opener.addheaders = [('User-agent','Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.56 Safari/535.11'), ('Accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'), ('Accept-Encoding', 'gzip,deflate,sdch'), ('Accept-Language', 'en-US,en;q=0.8'), ('Accept-Charset', 'ISO-8859-1,utf-8;q=0.7,*;q=0.3')] request = urllib2.Request(url, urlencode(data)) open("mycatpage.html", 'w').write(opener.open(request)) This has the same behavior...

    Read the article

  • Error 400 in urllib2 when using cookies

    - by Hempage
    I've had some success using a different method to load cookies, but now I'm wanting to use the cookielib.MozillaCookieJar method to open a cookies.txt file. Here's the snippet of code that does this. cookieJar=MozillaCookieJar() cookieJar.load(argv[2]) After creating an HTTPCookieProcessor opener, and installing it, whenever I use urlopen, I get an HTTP 400 error. However, if I don't use a CookieJar, the urlopen method succeeds (though the response doesn't contain the data I need). I'm not sure whether the cookies.txt file is malformed, or whether there's something else going on.

    Read the article

  • pyqt QObject: Cannot create children for a parent that is in a different thread

    - by memomk
    QObject: Cannot create children for a parent that is in a different thread. (Parent is QTextDocument(0x9919018), parent's thread is QThread(0x97331e0), current thread is flooderthread(0x97b4c10) error means ? am sorry because am new to pyqt here is the code : i know the code is finished yet but it should work i guess the problem is with myfun.log function... #! /usr/bin/python # -*- coding: utf-8 -*- import urllib, urllib2, itertools, threading, cookielib, Cookie, sys, time, hashlib, os from PyQt4 import QtCore, QtGui try: _fromUtf8 = QtCore.QString.fromUtf8 except AttributeError: _fromUtf8 = lambda s: s gui=QtGui.QApplication.processEvents texttoset="" class fun(): global texttoset def checkpassword(self): if ui.passwordcheck.isChecked()==True: return 1 else : return 0 def log(self, text): if text != False: firsttext=str(ui.console.toPlainText()) secondtext=firsttext+text+"\n" ui.console.setText(secondtext) log=open("log.log", "a") log.write(text+"\n") log.close() else : firsttext=str(ui.console.toPlainText()) secondtext=firsttext+texttoset+"\n" ui.console.setText(secondtext) log=open("log.log", "a") log.write(texttoset+"\n") log.close() def disable(self): MainWindow.setEnabled(False) pass def enable(self): MainWindow.setEnabled(True) pass def checkmethod(self): if ui.get.isChecked()==True: return 1 elif ui.post.isChecked()==True: return 2 else : return 0 def main(self): connecter() gui() f1.start() gui() time.sleep(3) gui() f2.start() gui() time.sleep(3) gui() f3.start() gui() time.sleep(3) gui() f4.start() gui() time.sleep(3) gui() f5.start() gui() self.sleep(3) gui() f6.start() gui() def killer(self): f1.terminate() f2.terminate() f3.terminate() f4.terminate() f5.terminate() f6.terminate() def close(self): self.killer() os.abort() sys.exit() myfun=fun() def connecter(): QtCore.QObject.connect(f1, QtCore.SIGNAL("log(bool)"), myfun.log) QtCore.QObject.connect(f1, QtCore.SIGNAL("enable()"), myfun.enable) QtCore.QObject.connect(f1, QtCore.SIGNAL("disable()"), myfun.disable) QtCore.QObject.connect(f2, QtCore.SIGNAL("log(bool)"), myfun.log) QtCore.QObject.connect(f2, QtCore.SIGNAL("enable()"), myfun.enable) QtCore.QObject.connect(f2, QtCore.SIGNAL("disable()"), myfun.disable) QtCore.QObject.connect(f3, QtCore.SIGNAL("log(bool)"), myfun.log) QtCore.QObject.connect(f3, QtCore.SIGNAL("enable()"), myfun.enable) QtCore.QObject.connect(f3, QtCore.SIGNAL("disable()"), myfun.disable) QtCore.QObject.connect(f4, QtCore.SIGNAL("log(bool)"), myfun.log) QtCore.QObject.connect(f4, QtCore.SIGNAL("enable()"), myfun.enable) QtCore.QObject.connect(f4, QtCore.SIGNAL("disable()"), myfun.disable) QtCore.QObject.connect(f5, QtCore.SIGNAL("log(bool)"), myfun.log) QtCore.QObject.connect(f5, QtCore.SIGNAL("enable()"), myfun.enable) QtCore.QObject.connect(f5, QtCore.SIGNAL("disable()"), myfun.disable) QtCore.QObject.connect(f6, QtCore.SIGNAL("log(bool)"), myfun.log) QtCore.QObject.connect(f6, QtCore.SIGNAL("enable()"), myfun.enable) QtCore.QObject.connect(f6, QtCore.SIGNAL("disable()"), myfun.disable) x=0 num=0 class flooderthread(QtCore.QThread): global texttoset def __init__(self, x, num): QtCore.QThread.__init__(self) self.x=x self.num=num def log(self, text): texttolog=str(text) time.sleep(1) self.emit(QtCore.SIGNAL("log(bool)"), False) time.sleep(2) def enable(self): time.sleep(1) self.emit(QtCore.SIGNAL("enable()")) def disable(self): time.sleep(1) self.emit(QtCore.SIGNAL("disable()")) def run(self): connecter() self.log("\n\n--------------------------------------------------new session-------------------------------------\n\n") itered=False gui() self.disable() gui() self.log("setting params...") param={ui.dataname1.text():ui.datavalue1.text(),ui.dataname3.text():ui.datavalue3.text(),ui.dataname3.text():ui.datavalue3.text(), } self.log("checking password...") if myfun.checkpassword()==1: itered=True self.log("password is true") else : self.log("password is null ") self.log("itered operation") self.log("setting url") url=str(ui.url.text()) if url[:4]!="http" and url[:3]!="ftp": self.log("url error exiting the whole function") self.log("please set a valide protocole!!") gui() self.enable() gui() return 1 pass else : self.log("valid url") gui() self.log("url is "+url) self.log("setting proxy") proxy="http://"+ui.proxyuser.text()+":"+ui.proxypass.text()+"@"+ui.proxyhost.text()+":"+ui.proxyport.text() self.log("proxy is "+proxy) gui() self.log("preparing params...") urlparam=urllib.urlencode(param) gui() self.log("params are "+urlparam) self.log("setting up headers...") header={'User-Agent':str(ui.useragent.toPlainText())} self.log("headers are "+ str(header)) self.log("setting up proxy handler..") proxyhandler=urllib2.ProxyHandler({"http":str(proxy)}) self.log("checking method") if myfun.checkmethod()==1: self.log("method is get..") self.log("setting request..") finalurl=url+urlparam gui() self.log("final url is"+finalurl) req=urllib2.Request(finalurl, None, headers) elif myfun.checkmethod()==2: self.log("method is post...") self.log("setting request..") finalurl=url gui() self.log("final url is "+finalurl) req=urllib2.Request(finalurl, urlparam, header) else : self.log("error has been accourded") self.log("please select a method!!") gui() self.log("exiting the whole functions") gui() self.enable() return 1 pass self.log("intilizing cookies..") c1=Cookie.SimpleCookie() c1[str(ui.cookiename1.text())]=str(ui.cookievalue1.text()) c1[str(ui.cookiename1.text())]['path']='/' c1[str(ui.cookiename2.text())]=str(ui.cookievalue2.text()) c1[str(ui.cookiename2.text())]['path']='/' c1[str(ui.cookiename3.text())]=str(ui.cookievalue3.text()) c1[str(ui.cookiename3.text())]['domain']=url c1[str(ui.cookiename3.text())]['path']='/' c1[str(ui.cookiename4.text())]=str(ui.cookievalue4.text()) c1[str(ui.cookiename4.text())]['domain']=url c1[str(ui.cookiename4.text())]['path']='/' self.log("cookies are.. :"+str(c1)) cj=cookielib.CookieJar() cj.set_cookie(c1) opener = urllib2.build_opener(proxyhandler, urllib2.HTTPCookieProcessor(cj)) self.log("insatlling opener") urllib2.install_opener(opener) self.log("setting the two operations....") if itered==Fasle: self.log("starting the flooding loop") gui() while true: try: gui() opener.open(req) except e: self.log("error connecting : "+e.reason) self.log("will continue....") continue gui() elif itered==True: pass f1=flooderthread(1, 1) f2=flooderthread(2, 2) f3=flooderthread(3, 3) f4=flooderthread(4, 4) f5=flooderthread(5, 5) f6=flooderthread(6, 6) class Ui_MainWindow(object): def setupUi(self, MainWindow): MainWindow.setObjectName(_fromUtf8("MainWindow")) MainWindow.setMinimumSize(QtCore.QSize(838, 500)) MainWindow.setMaximumSize(QtCore.QSize(838, 500)) MainWindow.setWindowTitle(QtGui.QApplication.translate("MainWindow", "memo flooder", None, QtGui.QApplication.UnicodeUTF8)) self.centralwidget = QtGui.QWidget(MainWindow) self.centralwidget.setObjectName(_fromUtf8("centralwidget")) self.console=QtGui.QTextEdit(self.centralwidget) self.console.setGeometry(10, 350, 800,130) self.console.setReadOnly(True) self.console.setObjectName("console") self.groupBox = QtGui.QGroupBox(self.centralwidget) self.groupBox.setGeometry(QtCore.QRect(30, 50, 71, 80)) self.groupBox.setTitle(QtGui.QApplication.translate("MainWindow", "method:", None, QtGui.QApplication.UnicodeUTF8)) self.groupBox.setObjectName(_fromUtf8("groupBox")) self.post = QtGui.QRadioButton(self.groupBox) self.post.setGeometry(QtCore.QRect(10, 20, 61, 22)) self.post.setText(QtGui.QApplication.translate("MainWindow", "post", None, QtGui.QApplication.UnicodeUTF8)) self.post.setChecked(True) self.post.setObjectName(_fromUtf8("post")) self.get = QtGui.QRadioButton(self.groupBox) self.get.setGeometry(QtCore.QRect(10, 50, 51, 22)) self.get.setText(QtGui.QApplication.translate("MainWindow", "get", None, QtGui.QApplication.UnicodeUTF8)) self.get.setObjectName(_fromUtf8("get")) self.url = QtGui.QLineEdit(self.centralwidget) self.url.setGeometry(QtCore.QRect(70, 20, 671, 27)) self.url.setInputMethodHints(QtCore.Qt.ImhUrlCharactersOnly) self.url.setObjectName(_fromUtf8("url")) self.groupBox_2 = QtGui.QGroupBox(self.centralwidget) self.groupBox_2.setGeometry(QtCore.QRect(110, 50, 371, 111)) self.groupBox_2.setTitle(QtGui.QApplication.translate("MainWindow", "data:", None, QtGui.QApplication.UnicodeUTF8)) self.groupBox_2.setObjectName(_fromUtf8("groupBox_2")) self.dataname1 = QtGui.QLineEdit(self.groupBox_2) self.dataname1.setGeometry(QtCore.QRect(20, 30, 101, 27)) self.dataname1.setObjectName(_fromUtf8("dataname1")) self.label = QtGui.QLabel(self.groupBox_2) self.label.setGeometry(QtCore.QRect(40, 10, 67, 17)) self.label.setText(QtGui.QApplication.translate("MainWindow", "name:", None, QtGui.QApplication.UnicodeUTF8)) self.label.setObjectName(_fromUtf8("label")) self.dataname2 = QtGui.QLineEdit(self.groupBox_2) self.dataname2.setGeometry(QtCore.QRect(130, 30, 113, 27)) self.dataname2.setObjectName(_fromUtf8("dataname2")) self.dataname3 = QtGui.QLineEdit(self.groupBox_2) self.dataname3.setGeometry(QtCore.QRect(250, 30, 113, 27)) self.dataname3.setObjectName(_fromUtf8("dataname3")) self.label_2 = QtGui.QLabel(self.groupBox_2) self.label_2.setGeometry(QtCore.QRect(40, 60, 67, 17)) self.label_2.setText(QtGui.QApplication.translate("MainWindow", "value:", None, QtGui.QApplication.UnicodeUTF8)) self.label_2.setObjectName(_fromUtf8("label_2")) self.datavalue1 = QtGui.QLineEdit(self.groupBox_2) self.datavalue1.setGeometry(QtCore.QRect(20, 80, 101, 27)) self.datavalue1.setObjectName(_fromUtf8("datavalue1")) self.datavalue2 = QtGui.QLineEdit(self.groupBox_2) self.datavalue2.setGeometry(QtCore.QRect(130, 80, 113, 27)) self.datavalue2.setObjectName(_fromUtf8("datavalue2")) self.datavalue3 = QtGui.QLineEdit(self.groupBox_2) self.datavalue3.setGeometry(QtCore.QRect(250, 80, 113, 27)) self.datavalue3.setObjectName(_fromUtf8("datavalue3")) self.groupBox_4 = QtGui.QGroupBox(self.centralwidget) self.groupBox_4.setGeometry(QtCore.QRect(670, 50, 151, 111)) self.groupBox_4.setTitle(QtGui.QApplication.translate("MainWindow", "password:", None, QtGui.QApplication.UnicodeUTF8)) self.groupBox_4.setObjectName(_fromUtf8("groupBox_4")) self.passname = QtGui.QLineEdit(self.groupBox_4) self.passname.setGeometry(QtCore.QRect(10, 30, 113, 27)) self.passname.setObjectName(_fromUtf8("passname")) self.passvalue = QtGui.QLineEdit(self.groupBox_4) self.passvalue.setGeometry(QtCore.QRect(10, 80, 113, 27)) self.passvalue.setObjectName(_fromUtf8("passvalue")) self.passwordcheck = QtGui.QCheckBox(self.centralwidget) self.passwordcheck.setGeometry(QtCore.QRect(670, 180, 97, 22)) self.passwordcheck.setText(QtGui.QApplication.translate("MainWindow", "password", None, QtGui.QApplication.UnicodeUTF8)) self.passwordcheck.setChecked(True) self.passwordcheck.setObjectName(_fromUtf8("passwordcheck")) self.groupBox_5 = QtGui.QGroupBox(self.centralwidget) self.groupBox_5.setGeometry(QtCore.QRect(29, 169, 441, 81)) self.groupBox_5.setTitle(QtGui.QApplication.translate("MainWindow", "proxy:", None, QtGui.QApplication.UnicodeUTF8)) self.groupBox_5.setObjectName(_fromUtf8("groupBox_5")) self.proxyhost = QtGui.QLineEdit(self.groupBox_5) self.proxyhost.setGeometry(QtCore.QRect(20, 30, 113, 27)) self.proxyhost.setObjectName(_fromUtf8("proxyhost")) self.proxyport = QtGui.QLineEdit(self.groupBox_5) self.proxyport.setGeometry(QtCore.QRect(140, 30, 51, 27)) self.proxyport.setInputMethodHints(QtCore.Qt.ImhDigitsOnly|QtCore.Qt.ImhPreferNumbers) self.proxyport.setObjectName(_fromUtf8("proxyport")) self.proxyuser = QtGui.QLineEdit(self.groupBox_5) self.proxyuser.setGeometry(QtCore.QRect(200, 30, 113, 27)) self.proxyuser.setObjectName(_fromUtf8("proxyuser")) self.proxypass = QtGui.QLineEdit(self.groupBox_5) self.proxypass.setGeometry(QtCore.QRect(320, 30, 113, 27)) self.proxypass.setObjectName(_fromUtf8("proxypass")) self.label_4 = QtGui.QLabel(self.groupBox_5) self.label_4.setGeometry(QtCore.QRect(100, 10, 67, 17)) self.label_4.setText(QtGui.QApplication.translate("MainWindow", "host", None, QtGui.QApplication.UnicodeUTF8)) self.label_4.setObjectName(_fromUtf8("label_4")) self.label_5 = QtGui.QLabel(self.groupBox_5) self.label_5.setGeometry(QtCore.QRect(150, 10, 67, 17)) self.label_5.setText(QtGui.QApplication.translate("MainWindow", "port", None, QtGui.QApplication.UnicodeUTF8)) self.label_5.setObjectName(_fromUtf8("label_5")) self.label_6 = QtGui.QLabel(self.groupBox_5) self.label_6.setGeometry(QtCore.QRect(200, 10, 67, 17)) self.label_6.setText(QtGui.QApplication.translate("MainWindow", "username", None, QtGui.QApplication.UnicodeUTF8)) self.label_6.setObjectName(_fromUtf8("label_6")) self.label_7 = QtGui.QLabel(self.groupBox_5) self.label_7.setGeometry(QtCore.QRect(320, 10, 67, 17)) self.label_7.setText(QtGui.QApplication.translate("MainWindow", "password", None, QtGui.QApplication.UnicodeUTF8)) self.label_7.setObjectName(_fromUtf8("label_7")) self.groupBox_6 = QtGui.QGroupBox(self.centralwidget) self.groupBox_6.setGeometry(QtCore.QRect(30, 260, 531, 91)) self.groupBox_6.setTitle(QtGui.QApplication.translate("MainWindow", "cookies:", None, QtGui.QApplication.UnicodeUTF8)) self.groupBox_6.setObjectName(_fromUtf8("groupBox_6")) self.cookiename1 = QtGui.QLineEdit(self.groupBox_6) self.cookiename1.setGeometry(QtCore.QRect(10, 20, 113, 27)) self.cookiename1.setObjectName(_fromUtf8("cookiename1")) self.cookiename2 = QtGui.QLineEdit(self.groupBox_6) self.cookiename2.setGeometry(QtCore.QRect(140, 20, 113, 27)) self.cookiename2.setObjectName(_fromUtf8("cookename2")) self.cookiename3 = QtGui.QLineEdit(self.groupBox_6) self.cookiename3.setGeometry(QtCore.QRect(270, 20, 113, 27)) self.cookiename3.setObjectName(_fromUtf8("cookiename3")) self.cookiename4 = QtGui.QLineEdit(self.groupBox_6) self.cookiename4.setGeometry(QtCore.QRect(390, 20, 113, 27)) self.cookiename4.setObjectName(_fromUtf8("cookiename4")) self.cookievalue1 = QtGui.QLineEdit(self.groupBox_6) self.cookievalue1.setGeometry(QtCore.QRect(10, 50, 113, 27)) self.cookievalue1.setObjectName(_fromUtf8("cookievalue1")) self.cookievalue2 = QtGui.QLineEdit(self.groupBox_6) self.cookievalue2.setGeometry(QtCore.QRect(140, 50, 113, 27)) self.cookievalue2.setObjectName(_fromUtf8("cookievalue2")) self.cookievalue3 = QtGui.QLineEdit(self.groupBox_6) self.cookievalue3.setGeometry(QtCore.QRect(270, 50, 113, 27)) self.cookievalue3.setObjectName(_fromUtf8("cookievalue3")) self.cookievalue4 = QtGui.QLineEdit(self.groupBox_6) self.cookievalue4.setGeometry(QtCore.QRect(390, 50, 113, 27)) self.cookievalue4.setObjectName(_fromUtf8("cookievalue4")) self.groupBox_7 = QtGui.QGroupBox(self.centralwidget) self.groupBox_7.setGeometry(QtCore.QRect(570, 260, 251, 80)) self.groupBox_7.setTitle(QtGui.QApplication.translate("MainWindow", "useragents:", None, QtGui.QApplication.UnicodeUTF8)) self.groupBox_7.setObjectName(_fromUtf8("groupBox_7")) self.useragent = QtGui.QTextEdit(self.groupBox_7) self.useragent.setGeometry(QtCore.QRect(10, 20, 211, 51)) self.useragent.setVerticalScrollBarPolicy(QtCore.Qt.ScrollBarAlwaysOn) self.useragent.setObjectName(_fromUtf8("useragent")) self.start = QtGui.QPushButton(self.centralwidget) self.start.setGeometry(QtCore.QRect(750, 20, 71, 27)) self.start.setText(QtGui.QApplication.translate("MainWindow", "start", None, QtGui.QApplication.UnicodeUTF8)) self.start.setObjectName(_fromUtf8("start")) self.label_3 = QtGui.QLabel(self.centralwidget) self.label_3.setGeometry(QtCore.QRect(30, 20, 67, 17)) self.label_3.setText(QtGui.QApplication.translate("MainWindow", "url :", None, QtGui.QApplication.UnicodeUTF8)) self.label_3.setObjectName(_fromUtf8("label_3")) MainWindow.setCentralWidget(self.centralwidget) QtCore.QObject.connect(self.start, QtCore.SIGNAL(_fromUtf8("clicked(bool)")), myfun.main) QtCore.QObject.connect(self.passwordcheck, QtCore.SIGNAL(_fromUtf8("clicked(bool)")), self.groupBox_4.setEnabled) QtCore.QMetaObject.connectSlotsByName(MainWindow) def __del__(): myfun.killer() os.abort() sys.exit() app = QtGui.QApplication(sys.argv) MainWindow = QtGui.QMainWindow() ui = Ui_MainWindow() ui.setupUi(MainWindow) myfun.log("\n\n--------------------------------------------------new session-------------------------------------\n\n") MainWindow.show() sys.exit(app.exec_())

    Read the article

  • Use the output of logs in the execution of a program

    - by myle
    When I try to create a specific object, the program crashes. However, I use a module (mechanize) which logs useful information just before the crash. If I had somehow this information available I could avoid it. Is there any way to use the information which is logged (when I use the function set_debug_redirects) during the normal execution of the program? Just to be a bit more specific, I try to emulate the login behavior in a webpage. The program crashes because it can't handle a specific Following HTTP-EQUIV=REFRESH to <omitted_url>. Given this url, which is available in the logs but not as part of the exception which is thrown, I could visit this page and complete successfully the login process. Any other suggestions that may solve the problem are welcomed. It follows the code so far. SERVICE_LOGIN_BOX_URL = "https://www.google.com/accounts/ServiceLoginBox?service=adsense&ltmpl=login&ifr=true&rm=hide&fpui=3&nui=15&alwf=true&passive=true&continue=https%3A%2F%2Fwww.google.com%2Fadsense%2Flogin-box-gaiaauth&followup=https%3A%2F%2Fwww.google.com%2Fadsense%2Flogin-box-gaiaauth&hl=en_US" def init_browser(): # Browser br = mechanize.Browser() # Cookie Jar cj = cookielib.LWPCookieJar() br.set_cookiejar(cj) # Browser options br.set_handle_equiv(True) br.set_handle_gzip(False) br.set_handle_redirect(True) br.set_handle_referer(True) br.set_handle_robots(True) br.set_handle_refresh(mechanize._http.HTTPRefreshProcessor(), max_time=30.0, honor_time=False) # Want debugging messages? #br.set_debug_http(True) br.set_debug_redirects(True) #br.set_debug_responses(True) return br def adsense_login(login, password): br = init_browser() r = br.open(SERVICE_LOGIN_BOX_URL) html = r.read() # Select the first (index zero) form br.select_form(nr=0) br.form['Email'] = login br.form['Passwd'] = password br.submit() req = br.click_link(text='click here to continue') try: # this is where it crashes br.open(req) except HTTPError, e: sys.exit("post failed: %d: %s" % (e.code, e.msg)) return br

    Read the article

  • Python - Problems using mechanize to log into a difficult website

    - by user1781599
    × 139886 I am trying to log in to betfair.com by using mechanize. I have tried several ways but it always fail. This is the code I have developed so far, can anyone help me to identify what is wrong with it and how I can improve it to log into my betfair account? Thanks, import cookielib import urllib import urllib2 from BeautifulSoup import BeautifulSoup import mechanize from mechanize import Browser import re bf_username_name = "username" bf_password_name = "password" bf_form_name = "loginForm" bf_username = "xxxxx" bf_password = "yyyyy" urlLogIn = "http://www.betfair.com/" accountUrl = "https://myaccount.betfair.com/account/home?rlhm=0&" # This url I will use to verify if log in has been successful br = mechanize.Browser(factory=mechanize.RobustFactory()) br.addheaders = [("User-Agent","Mozilla/5.0 (Macintosh; Intel Mac OS X 10_5_8) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.90 Safari/537.1")] br.open(urlLogIn) br.select_form(nr=0) print br.form br.form[bf_username_name] = bf_username br.form[bf_password_name] = bf_password print br.form #just to check username and psw have been recorded correctly responseSubmit = br.submit() response = br.open(accountUrl) text_file = open("LogInResponse.html", "w") text_file.write(responseSubmit.read()) #this file should show the home page with me logged in, but it show home page as if I was not logged it text_file.close() text_file = open("Account.html", "w") text_file.write(response.read()) #this file should show my account page, but it should a pop up with an error text_file.close()

    Read the article

1