How to handle response encoding from urllib.request.urlopen()

python regex encoding urllib
kryptobs2000 picture kryptobs2000 · Feb 13, 2011 · Viewed 92.2k times · Source

I'm trying to search a webpage using regular expressions, but I'm getting the following error:

TypeError: can't use a string pattern on a bytes-like object

I understand why, urllib.request.urlopen() returns a bytestream and so, at least I'm guessing, re doesn't know the encoding to use. What am I supposed to do in this situation? Is there a way to specify the encoding method in a urlrequest maybe or will I need to re-encode the string myself? If so what am I looking to do, I assume I should read the encoding from the header info or the encoding type if specified in the html and then re-encode it to that?

Answer

Ivan Klass picture Ivan Klass · Oct 3, 2013

As for me, the solution is as following (python3):

resource = urllib.request.urlopen(an_url)
content =  resource.read().decode(resource.headers.get_content_charset())

Related questions

Does "\d" in regex mean a digit?

I found that in 123, \d matches 1 and 3 but not 2. I was wondering if \d matches a digit satisfying what kind of requirement? I am talking about Python style regex. Regular expression plugin in Gedit is using Python style regex. I …

Read More
Python - Replace non-ascii character in string (»)

I need to replace in a string the character "»" with a whitespace, but I still get an error. This is the code I use: # -*- coding: utf-8 -*- from bs4 import BeautifulSoup # other code soup = BeautifulSoup(data, 'lxml') …

Read More
How to extract numbers from a string in Python?

I would extract all the numbers contained in a string. Which is the better suited for the purpose, regular expressions or the isdigit() method? Example: line = "hello 12 hi 89" Result: [12, 89]

Read More