Loading url with cyrillic symbols
Posted
by Ockonal
on Stack Overflow
See other posts from Stack Overflow
or by Ockonal
Published on 2010-05-14T15:19:09Z
Indexed on
2010/05/14
15:44 UTC
Read the original article
Hit count: 295
Hi guys, I have to load some url with cyrillic symbols. My script should work with this:
If I'll use this in browser it would replaced into normal symbols, but urllib code fails with 404 error. How to decode correctly this url?
When I'm using that url directly in code, like address = 'that address', it works perfect. But I used parsing page for getting this url. I have a list of urls which contents cyrillic. Maybe they have uncorrect encoding? Here is more code:
requestData = urllib2.Request( %SOME_ADDRESS%, None, {"User-Agent": user_agent})
requestHandler = pageHandler.open(requestData)
pageData = requestHandler.read().decode('utf-8')
soupHandler = BeautifulSoup(pageData)
topicLinks = []
for postBlock in soupHandler.findAll('a', href=re.compile('%SOME_REGEXP%')):
topicLinks.append(postBlock['href'])
postAddress = choice(topicLinks)
postRequestData = urllib2.Request(postAddress, None, {"User-Agent": user_agent})
postHandler = pageHandler.open(postRequestData)
postData = postHandler.read()
File "/usr/lib/python2.6/urllib2.py", line 518, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 404: Not Found
© Stack Overflow or respective owner