Search Results

Search found 166 results on 7 pages for 'urllib'.

Page 1/7 | 1 2 3 4 5 6 7  | Next Page >

  • urllib open - how to control the number of retries

    - by user1641071
    how can i control the number of retries of the "opener.open"? for example, in the following code, it will send about 6 "GET" HTTP requests (i saw it in the Wireshark sniffer) before it goes to the " except urllib.error.URLError" success/no-success lines. password_mgr = urllib.request.HTTPPasswordMgrWithDefaultRealm() password_mgr.add_password(None,url, username, password) handler = urllib.request.HTTPBasicAuthHandler(password_mgr) opener = urllib.request.build_opener(handler) try: resp = opener.open(url,None,1) except urllib.error.URLError as e: print ("no success") else: print ("success!")

    Read the article

  • Problem with python urllib

    - by mudder
    I'm getting an error when ever I try to pull down a web page with urllib.urlopen. I've disabled windows firewall and my AV so its not that. I can access the pages in my browser. I even reinstalled python to rule out it being a broken urllib. Any help would be greatly appreciated. >>> import urllib >>> h = urllib.urlopen("http://www.google.com").read() Traceback (most recent call last): File "<pyshell#1>", line 1, in <module> h = urllib.urlopen("http://www.google.com").read() File "C:\Python26\lib\urllib.py", line 86, in urlopen return opener.open(url) File "C:\Python26\lib\urllib.py", line 205, in open return getattr(self, name)(url) File "C:\Python26\lib\urllib.py", line 344, in open_http h.endheaders() File "C:\Python26\lib\httplib.py", line 904, in endheaders self._send_output() File "C:\Python26\lib\httplib.py", line 776, in _send_output self.send(msg) File "C:\Python26\lib\httplib.py", line 735, in send self.connect() File "C:\Python26\lib\httplib.py", line 716, in connect self.timeout) File "C:\Python26\lib\socket.py", line 514, in create_connection raise error, msg IOError: [Errno socket error] [Errno 10061] No connection could be made because the target machine actively refused it >>>

    Read the article

  • Problem with urllib

    - by Eva
    I wrote this code: import urllib proxies = {'http': 'http://112.65.135.54:8080/'} opener = urllib.FancyURLopener(proxies) r = opener.open("http://www.python.org/") print r.read() and when I execute it this program works fine, and send for me source code of python.org But when i use this: import urllib proxies = {'http': 'http://80.176.245.196:1080/'} opener = urllib.FancyURLopener(proxies) r = opener.open("http://www.python.org/") print r.read() this program does not send me the source code of python.org What am I going to do?

    Read the article

  • Python urllib.urlopen IOError

    - by Michael
    So I have the following lines of code in a function sock = urllib.urlopen(url) html = sock.read() sock.close() and they work fine when I call the function by hand. However, when I call the function in a loop (using the same urls as earlier) I get the following error: > Traceback (most recent call last): File "./headlines.py", line 256, in <module> main(argv[1:]) File "./headlines.py", line 37, in main write_articles(headline, output_folder + "articles_" + term +"/") File "./headlines.py", line 232, in write_articles print get_blogs(headline, 5) File "/Users/michaelnussbaum08/Documents/College/Sophmore_Year/Quarter_2/Innovation/Headlines/_code/get_content.py", line 41, in get_blogs sock = urllib.urlopen(url) File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/urllib.py", line 87, in urlopen return opener.open(url) File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/urllib.py", line 203, in open return getattr(self, name)(url) File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/urllib.py", line 314, in open_http if not host: raise IOError, ('http error', 'no host given') IOError: [Errno http error] no host given Any ideas?

    Read the article

  • Python interface to PayPal - urllib.urlencode non-ASCII characters failing

    - by krys
    I am trying to implement PayPal IPN functionality. The basic protocol is as such: The client is redirected from my site to PayPal's site to complete payment. He logs into his account, authorizes payment. PayPal calls a page on my server passing in details as POST. Details include a person's name, address, and payment info etc. I need to call a URL on PayPal's site internally from my processing page passing back all the params that were passed in abovem and an additional one called 'cmd' with a value of '_notify-validate'. When I try to urllib.urlencode the params which PayPal has sent to me, I get a: While calling send_response_to_paypal. Traceback (most recent call last): File "<snip>/account/paypal/views.py", line 108, in process_paypal_ipn verify_result = send_response_to_paypal(params) File "<snip>/account/paypal/views.py", line 41, in send_response_to_paypal params = urllib.urlencode(params) File "/usr/local/lib/python2.6/urllib.py", line 1261, in urlencode v = quote_plus(str(v)) UnicodeEncodeError: 'ascii' codec can't encode character u'\ufffd' in position 9: ordinal not in range(128) I understand that urlencode does ASCII encoding, and in certain cases, a user's contact info can contain non-ASCII characters. This is understandable. My question is, how do I encode non-ASCII characters for POSTing to a URL using urllib2.urlopen(req) (or other method) Details: I read the params in PayPal's original request as follows (the GET is for testing): def read_ipn_params(request): if request.POST: params= request.POST.copy() if "ipn_auth" in request.GET: params["ipn_auth"]=request.GET["ipn_auth"] return params else: return request.GET.copy() The code I use for sending back the request to PayPal from the processing page is: def send_response_to_paypal(params): params['cmd']='_notify-validate' params = urllib.urlencode(params) req = urllib2.Request(PAYPAL_API_WEBSITE, params) req.add_header("Content-type", "application/x-www-form-urlencoded") response = urllib2.urlopen(req) status = response.read() if not status == "VERIFIED": logging.warn("PayPal cannot verify IPN responses: " + status) return False return True Obviously, the problem only arises if someone's name or address or other field used for the PayPal payment does not fall into the ASCII range.

    Read the article

  • python urllib post question

    - by paul
    hello ALL im making some simple python post script but it not working well. there is 2 part to have to login. first login is using 'http://mybuddy.buddybuddy.co.kr/userinfo/UserInfo.asp' this one. and second login is using 'http://user.buddybuddy.co.kr/usercheck/UserCheckPWExec.asp' i can login first login page, but i couldn't login second page website. and return some error 'illegal access' such like . i heard this is related with some cooke but i don't know how to implement to resolve this problem. if anyone can help me much appreciated!! Thanks! import re,sys,os,mechanize,urllib,time import datetime,socket params = urllib.urlencode({'ID':'ph896011', 'PWD':'pk1089' }) rq = mechanize.Request("http://mybuddy.buddybuddy.co.kr/userinfo/UserInfo.asp", params) rs = mechanize.urlopen(rq) data = rs.read() logged_fail = r';history.back();</script>' in data if not logged_fail: print 'login success' try: params = urllib.urlencode({'PASSWORD':'pk1089'}) rq = mechanize.Request("http://user.buddybuddy.co.kr/usercheck/UserCheckPWExec.asp", params ) rs = mechanize.urlopen(rq) data = rs.read() print data except: print 'error'

    Read the article

  • Cookies with urllib

    - by CMC
    This will probably seem like a really simple question, and I am quite confused as to why this is so difficult for me. I would like to write a function that takes three inputs: [url, data, cookies] that will use urllib (not urllib2) to get the contents of the requested url. I figured it'd be simple, so I wrote the following: def fetch(url, data = None, cookies = None): if isinstance(data, dict): data = urllib.urlencode(data) if isinstance(cookies, dict): # TODO: find a better way to do this cookies = "; ".join([str(key) + "=" + str(cookies[key]) for key in cookies]) opener = urllib.FancyURLopener() opener.addheader("Cookie", cookies) obj = opener.open(url, data) result = obj.read() obj.close() return result This doesn't work, as far as I can tell (can anyone confirm that?) and I'm stumped.

    Read the article

  • Python: ImportError no module named urllib

    - by Yury Lifshits
    I just rented a VPS from Linode, it has python2.5 and ubuntu 8.04 When I ho to python shell python import urllib I get ImportError: No module named urllib What can be the reason? How can I add this module to python? Isn't it prepackaged with the basic version? Can it be pythonpath problem? How I can test pythonpath?

    Read the article

  • Python urllib.urlopen() call doesn't work with a URL that a browser accepts

    - by Charles Anderson
    If I point Firefox at http://bitbucket.org/tortoisehg/stable/wiki/Home/ReleaseNotes, I get a page of HTML. But if I try this in Python: import urllib site = 'http://bitbucket.org/tortoisehg/stable/wiki/Home/ReleaseNotes' req = urllib.urlopen(site) text = req.read() I get the following: 500 Internal Server Error The server encountered an internal error or misconfiguration and was unable to complete your request. What am I doing wrong?

    Read the article

  • Python: fetching SVG file using urllib is returning binary when I need ASCII

    - by Drew Dara-Abrams
    I'm using urllib (in Python) to fetch an SVG file: import urllib urllib.urlopen('http://alpha.vectors.cloudmade.com/BC9A493B41014CAABB98F0471D759707/-122.2487,37.87588,-122.265823,37.868054?styleid=1&viewport=400x231').read() which produces output of the sort: xb6\xf6\x00\xb3\xfb2\xff\xda\xc5\xf2\xc2\x14\xef\xcd\x82\x0b\xdbU\xb0\x81\xcaF\xd8\x1a\xf6\xdf[i)\xba\xcf\x80\xab\xd6\x8c\xe3l_\xe7\n\xed2,\xbdm\xa0_|\xbb\x12\xff\xb6\xf8\xda\xd9\xc3\xd9\t\xde\x9a\xf8\xae\xb3T\xa3\r`\x8a\x08!T\xfb8\x92\x95\x0c\xdd\x8b!\x02P\xea@\x98\x1c^\xc7\xda\\\xec\xe3\xe1\xbe,0\xcd\xbeZ~\x92\xb3\xfa\xdd\xfcbyu\xb8\x83\xbb\xbdS\x0f\x82\x0b\xfe\xf5_\xdawn\xff\xef_\xff\xe5\xfa\x1f?\xbf\xffoZ\x0f\x8b\xbfV\xf4\x04\x00' when I was expecting more like this: <?xml version='1.0' encoding='UTF-8'?> <svg xmlns="http://www.w3.org/2000/svg" xmlns:cm="http://cloudmade.com/" width="400" height="231"> <rect width="100%" height="100%" fill="#eae8dd" opacity="1"/> <g transform="scale(0.209849975856)"> <g transform="translate(13610569, 4561906)" flood-opacity="0.1" flood-color="grey"> <path d="M -13610027.720000000670552 -4562403.660000000149012 I guess this is an issue of binary vs. ASCII. Can anyone help me (a Python newbie) with the appropriate conversion so that I can get on with parsing and manipulating the SVG code?

    Read the article

  • Downloading a picture via urllib and python.

    - by Mike
    So I'm trying to make a Python script that downloads webcomics and puts them in a folder on my desktop. I've found a few similar programs on here that do something similar, but nothing quite like what I need. The one that I found most similar is right here (http://bytes.com/topic/python/answers/850927-problem-using-urllib-download-images). I tried using this code: >>> import urllib >>> image = urllib.URLopener() >>> image.retrieve("http://www.gunnerkrigg.com//comics/00000001.jpg","00000001.jpg") ('00000001.jpg', <httplib.HTTPMessage instance at 0x1457a80>) I then searched my computer for a file "00000001.jpg", but all I found was the cached picture of it. I'm not even sure it saved the file to my computer. Once I understand how to get the file downloaded, I think I know how to handle the rest. Essentially just use a for loop and split the string at the '00000000'.'jpg' and increment the '00000000' up to the largest number, which I would have to somehow determine. Any reccomendations on the best way to do this or how to download the file correctly? Thanks!

    Read the article

  • Urllib's urlopen broken on some sites (StackApps api)

    - by Edan Maor
    I'm using urllib2's urlopen function to try and get a JSON result from the StackOverflow api. The code I'm using: >>> import urllib2 >>> conn = urllib2.urlopen("http://api.stackoverflow.com/0.8/users/") >>> conn.readline() The result I'm getting: '\x1f\x8b\x08\x00\x00\x00\x00\x00\x04\x00\xed\xbd\x07`\x1cI\x96%&/m\xca{\x7fJ\... I'm fairly new to urllib, but this doesn't seem like the result I should be getting. I've tried it in other places and I get what I expect (the same as visiting the address with a browser gives me: a JSON object). Using urlopen on other sites (e.g. "http://google.com") works fine, and gives me actual html. I've also tried using urllib and it gives the same result. I'm pretty stuck, not even knowing where to look to solve this problem. Any ideas?

    Read the article

  • Trouble with encoding and urllib

    - by Ockonal
    Hello, I'm loading web-page using urllib. Ther eis russian symbols, but page encoding is 'utf-8' 1 pageData = unicode(requestHandler.read()).decode('utf-8') UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 262: ordinal not in range(128) 2 pageData = requestHandler.read() soupHandler = BeautifulSoup(pageData) print soupHandler.findAll(...) UnicodeEncodeError: 'ascii' codec can't encode characters in position 340-345: ordinal not in range(128)

    Read the article

  • Difference between Python urllib.urlretrieve() and wget

    - by jrdioko
    I am trying to retrieve a 500mb file using Python, and I have a script which uses urllib.urlretrieve(). There seems to some network problem between me and the download site, as this call consistently hangs and fails to complete. However, using wget to retrieve the file tends to work without problems. What is the difference between urlretrieve() and wget that could cause this difference?

    Read the article

  • making urllib request in Python from the client side

    - by mridang
    Hi Guys, I've written a Python application that makes web requests using the urllib2 library after which it scrapes the data. I could deploy this as a web application which means all urllib2 requests go through my web-server. This leads to the danger of the server's IP being banned due to the high number of web requests for many users. The other option is to create an desktop application which I don't want to do. Is there any way I could deploy my application so that I can get my web-requests through the client side. One way was to use Jython to create an applet but I've read that Java applets can only make web-requests to the server it is deployed on and the only way to to circumvent this is to create a server side proxy which leads us back to the problem of the server's ip getting banned. This might sounds sound like and impossible situation and I'll probably end up creating a desktop application but I thought I'd ask if anyone knew of an alternate solution. Thanks.

    Read the article

  • What can I do with urllib?

    - by Aerovistae
    I grasp that it can fetch internet resources, but I'm not really fully appreciating what that means. I was doing the challenges at pythonchallenge.com, and I got past the one that requires urllib to fetch the page and read the source, but I don't see how this is useful quite yet. And examples are kind of lacking in diversity. What else can it do, or how can I use this to my advantage? (Besides building a web browser...)

    Read the article

  • Urllib's urlopen breaking on some sites (e.g. StackApps api)

    - by Edan Maor
    I'm using urllib2's urlopen function to try and get a JSON result from the StackOverflow api. The code I'm using: >>> import urllib2 >>> conn = urllib2.urlopen("http://api.stackoverflow.com/0.8/users/") >>> conn.readline() The result I'm getting: '\x1f\x8b\x08\x00\x00\x00\x00\x00\x04\x00\xed\xbd\x07`\x1cI\x96%&/m\xca{\x7fJ\... I'm fairly new to urllib, but this doesn't seem like the result I should be getting. I've tried it in other places and I get what I expect (the same as visiting the address with a browser gives me: a JSON object). Using urlopen on other sites (e.g. "http://google.com") works fine, and gives me actual html. I've also tried using urllib and it gives the same result. I'm pretty stuck, not even knowing where to look to solve this problem. Any ideas?

    Read the article

  • Urllib's urlopen broken on some sites (e.g. StackApps api)

    - by Edan Maor
    I'm using urllib2's urlopen function to try and get a JSON result from the StackOverflow api. The code I'm using: >>> import urllib2 >>> conn = urllib2.urlopen("http://api.stackoverflow.com/0.8/users/") >>> conn.readline() The result I'm getting: '\x1f\x8b\x08\x00\x00\x00\x00\x00\x04\x00\xed\xbd\x07`\x1cI\x96%&/m\xca{\x7fJ\... I'm fairly new to urllib, but this doesn't seem like the result I should be getting. I've tried it in other places and I get what I expect (the same as visiting the address with a browser gives me: a JSON object). Using urlopen on other sites (e.g. "http://google.com") works fine, and gives me actual html. I've also tried using urllib and it gives the same result. I'm pretty stuck, not even knowing where to look to solve this problem. Any ideas?

    Read the article

  • Urllib's urlopen breaking on some sites (e.g. StackApps api): returns garbage results

    - by Edan Maor
    I'm using urllib2's urlopen function to try and get a JSON result from the StackOverflow api. The code I'm using: >>> import urllib2 >>> conn = urllib2.urlopen("http://api.stackoverflow.com/0.8/users/") >>> conn.readline() The result I'm getting: '\x1f\x8b\x08\x00\x00\x00\x00\x00\x04\x00\xed\xbd\x07`\x1cI\x96%&/m\xca{\x7fJ\... I'm fairly new to urllib, but this doesn't seem like the result I should be getting. I've tried it in other places and I get what I expect (the same as visiting the address with a browser gives me: a JSON object). Using urlopen on other sites (e.g. "http://google.com") works fine, and gives me actual html. I've also tried using urllib and it gives the same result. I'm pretty stuck, not even knowing where to look to solve this problem. Any ideas?

    Read the article

  • Python 3, urllib ... Reset Connection Possible?

    - by Rhys
    In the larger scale of my program the goal of the below code is to filter out all dynamic html in a web-page source code code snippet: try: deepreq3 = urllib.request.Request(deepurl3) deepreq3.add_header("User-Agent","etc......") deepdata3 = urllib.request.urlopen(deepurl3).read().decode("utf8", 'ignore') The following code is looped 3 times in order to identify whether the target web-page is Dynamic (source code is changed at intervals) or not. If the page IS dynamic, the above code loops another 15 times and attempts to filter out the dynamic content. QUESTION: While this filtering method works 80% of the time, some pages will reload ALL 15 times and STILL contain dynamic code. HOWEVER. If I manually close down the Python Shell and re-execute my program, the dynamic html that my 'refresh-page method' could not shake off is no longer there ... it's been replaced with new dynamic html that my 'refresh-page method' cannot shake off. So I need to know, what is going on here? How is re-running my program causing the dynamic content of a page to change. AND, is there any way, any 'reset connection' command I can use to recreate this ... without manually restarting my app. Thanks for your response.

    Read the article

  • Why can't I get Python's urlopen() method to work?

    - by froadie
    Why isn't this simple Python code working? import urllib file = urllib.urlopen('http://www.google.com') print file.read() This is the error that I get: Traceback (most recent call last): File "C:\workspace\GarchUpdate\src\Practice.py", line 26, in <module> file = urllib.urlopen('http://www.google.com') File "C:\Python26\lib\urllib.py", line 87, in urlopen return opener.open(url) File "C:\Python26\lib\urllib.py", line 206, in open return getattr(self, name)(url) File "C:\Python26\lib\urllib.py", line 345, in open_http h.endheaders() File "C:\Python26\lib\httplib.py", line 892, in endheaders self._send_output() File "C:\Python26\lib\httplib.py", line 764, in _send_output self.send(msg) File "C:\Python26\lib\httplib.py", line 723, in send self.connect() File "C:\Python26\lib\httplib.py", line 704, in connect self.timeout) File "C:\Python26\lib\socket.py", line 514, in create_connection raise error, msg IOError: [Errno socket error] [Errno 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond I've tried it with several different pages but I can never get the urlopen method to execute correctly.

    Read the article

  • Retrieving information with Python's urllib from a page that is done via __doPostBack()?

    - by Omar
    I'm trying to parse a page that has different sections that are loaded with a Javascript __doPostBack() function. An example of a link is: javascript:__doPostBack('ctl00$cphMain$ucOemSchPicker$dlSch$ctl03$btnSch','') As soon as this is clicked, the browser doesn't fetch a new URL but a section of webpage is updated to reflect new information. What would I pass into a urllib function to complete the operation?

    Read the article

1 2 3 4 5 6 7  | Next Page >