Python urllib3 and how to handle cookie support?

Posted by bigredbob on Stack Overflow See other posts from Stack Overflow or by bigredbob
Published on 2010-03-11T06:03:10Z Indexed on 2010/05/18 2:10 UTC
Read the original article Hit count: 292

Filed under:
|

So I'm looking into urllib3 because it has connection pooling and is thread safe (so performance is better, especially for crawling), but the documentation is... minimal to say the least. urllib2 has build_opener so something like:

#!/usr/bin/python
import cookielib, urllib2
cj = cookielib.CookieJar()
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))
r = opener.open("http://example.com/")

But urllib3 has no build_opener method, so the only way I have figured out so far is to manually put it in the header:

#!/usr/bin/python
import urllib3
http_pool = urllib3.connection_from_url("http://example.com")
myheaders = {'Cookie':'some cookie data'}
r = http_pool.get_url("http://example.org/", headers=myheaders)

But I am hoping there is a better way and that one of you can tell me what it is. Also can someone tag this with "urllib3" please.

© Stack Overflow or respective owner

Related posts about python

Related posts about urllib3