How to work with unicode in Python

python string unicode replace unicode-string
PyNEwbie picture PyNEwbie · Apr 15, 2009 · Viewed 18.6k times · Source

I am trying to clean all of the HTML out of a string so the final output is a text file. I have some some research on the various 'converters' and am starting to lean towards creating my own dictionary for the entities and symbols and running a replace on the string. I am considering this because I want to automate the process and there is a lot of variability in the quality of the underlying html. To begin comparing the speed of my solution and one of the alternatives for example pyparsing I decided to test replace of \xa0 using the string method replace. I get a 

UnicodeDecodeError: 'ascii' codec can't decode byte 0xa0 in position 0: ordinal not in range(128)

The actual line of code was 

s=unicodestring.replace('\xa0','')

Anyway-I decided that I needed to preface it with an r so I ran this line of code:

s=unicodestring.replace(r'\xa0','')

It runs without error but I when I look at a slice of s I see that the \xaO is still there

Answer

z33m picture z33m · Apr 15, 2009

may be you should be doing

s=unicodestring.replace(u'\xa0',u'')

Related questions

Convert a Unicode string to a string in Python (containing extra symbols)

How do you convert a Unicode string (containing extra characters like £ $, etc.) into a Python string?

Read More
What does the 'b' character do in front of a string literal?

Apparently, the following is the valid syntax: my_string = b'The string' I would like to know: What does this b character in front of the string mean? What are the effects of using it? What are appropriate situations to use …

Read More
Usage of unicode() and encode() functions in Python

I have a problem with encoding of the path variable and inserting it to the SQLite database. I tried to solve it with encode("utf-8") function which didn't help. Then I used unicode() function which gives me type unicode. print …

Read More