Loading url with cyrillic symbols

Posted by Ockonal on Stack Overflow See other posts from Stack Overflow or by Ockonal
Published on 2010-05-14T15:19:09Z Indexed on 2010/05/14 15:44 UTC
Read the original article Hit count: 295

Filed under:
|
|
|

Hi guys, I have to load some url with cyrillic symbols. My script should work with this:

http://wincode.org/%D0%BF%D1%80%D0%BE%D0%B3%D1%80%D0%B0%D0%BC%D0%BC%D0%B8%D1%80%D0%BE%D0%B2%D0%B0%D0%BD%D0%B8%D0%B5/

If I'll use this in browser it would replaced into normal symbols, but urllib code fails with 404 error. How to decode correctly this url?


When I'm using that url directly in code, like address = 'that address', it works perfect. But I used parsing page for getting this url. I have a list of urls which contents cyrillic. Maybe they have uncorrect encoding? Here is more code:

requestData = urllib2.Request( %SOME_ADDRESS%, None,  {"User-Agent": user_agent})
requestHandler = pageHandler.open(requestData)

pageData = requestHandler.read().decode('utf-8')
soupHandler = BeautifulSoup(pageData)

topicLinks = []
for postBlock in soupHandler.findAll('a', href=re.compile('%SOME_REGEXP%')):
    topicLinks.append(postBlock['href'])

postAddress = choice(topicLinks)

postRequestData = urllib2.Request(postAddress, None,  {"User-Agent": user_agent})
postHandler = pageHandler.open(postRequestData)
postData = postHandler.read()

  File "/usr/lib/python2.6/urllib2.py", line 518, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 404: Not Found

© Stack Overflow or respective owner

Related posts about urllib

Related posts about python