Using httplib2 in python 3 properly? (Timeout problems)

Posted by Sho Minamimoto on Stack Overflow See other posts from Stack Overflow or by Sho Minamimoto
Published on 2010-01-30T06:52:54Z Indexed on 2010/03/24 1:03 UTC
Read the original article Hit count: 450

Filed under:
|
|
|

Hey, first time post, I'm really stuck on httplib2. I've been reading up on it from diveintopython3.org, but it mentions nothing about a timeout function. I look up the documentation, but the only thing I see is an ability to put a timeout int but there are no units specified (seconds? milliseconds? What's the default if None?) This is what I have (I also have code to check what the response is and try again, but it's never tried more than once)

h = httplib2.Http('.cache', timeout=None)
for url in list:
    response, content = h.request(url)
    more stuff...

So the Http object stays around until some arbitrary time, but I'm downloading a ton of pages from the same server, and after a while, it hangs on getting a page. No errors are thrown, the thing just hangs at a page. So then I try:

h = httplib2.Http('.cache', timeout=None)
for url in list:
    try:
        response, content = h.request(url)
    except:
        h = httplib2.Http('.cache', timeout=None)
    more stuff...

But then it recreates another Http object every time (goes down the 'except' path)...I dont understand how to keep getting with the same object, until it expires and I make another. Also, is there a way to set a timeout on an individual request?

Thanks for the help!

© Stack Overflow or respective owner

Related posts about python

Related posts about httplib2