UnicodeEncodeError: 'charmap' codec can't encode - character maps to <undefined>, print function

Question

I am writing a Python (Python 3.3) program to send some data to a webpage using POST method. Mostly for debugging process I am getting the page result and displaying it on the screen using print() function.

The code is like this:

conn.request("POST", resource, params, headers)
response = conn.getresponse()
print(response.status, response.reason)
data = response.read()
print(data.decode('utf-8'));

the HTTPResponse .read() method returns a bytes element encoding the page (which is a well formated UTF-8 document) It seemed okay until I stopped using IDLE GUI for Windows and used the Windows console instead. The returned page has a U+2014 character (em-dash) which the print function translates well in the Windows GUI (I presume Code Page 1252) but does not in the Windows Console (Code Page 850). Given the strict default behavior I get the following error:

UnicodeEncodeError: 'charmap' codec can't encode character '\u2014' in position 10248: character maps to <undefined>

I could fix it using this quite ugly code:

print(data.decode('utf-8').encode('cp850','replace').decode('cp850'))

Now it replace the offending character "—" with a ?. Not the ideal case (a hyphen should be a better replacement) but good enough for my purpose.

There are several things I do not like from my solution.

1. The code is ugly with all that decoding, encoding, and decoding.
2. It solves the problem for just this case. If I port the program for a system using some other encoding (latin-1, cp437, back to cp1252, etc.) it should recognize the target encoding. It does not. (for instance, when using again the IDLE GUI, the emdash is also lost, which didn't happen before)
3. It would be nicer if the emdash translated to a hyphen instead of a interrogation bang.

The problem is not the emdash (I can think of several ways to solve that particularly problem) but I need to write robust code. I am feeding the page with data from a database and that data can come back. I can anticipate many other conflicting cases: an 'Á' U+00c1 (which is possible in my database) could translate into CP-850 (DOS/Windows Console encodign for Western European Languages) but not into CP-437 (encoding for US English, which is default in many Windows instalations).

So, the question:

Is there a nicer solution that makes my code agnostic from the output interface encoding?

Answer 1

I see three solutions to this:

1. Change the output encoding, so it will always output UTF-8. See e.g. Setting the correct encoding when piping stdout in Python, but I could not get these example to work.

2. Following example code makes the output aware of your target charset.

   # -*- coding: utf-8 -*-
   import sys
   
   print sys.stdout.encoding
   print u"Stöcker".encode(sys.stdout.encoding, errors='replace')
   print u"Стоескер".encode(sys.stdout.encoding, errors='replace')
   

   This example properly replaces any non-printable character in my name with a question mark.

   If you create a custom print function, e.g. called myprint, using that mechanisms to encode output properly you can simply replace print with myprint whereever necessary without making the whole code look ugly.

3. Reset the output encoding globally at the begin of the software:

   The page http://www.macfreek.nl/memory/Encoding_of_Python_stdout has a good summary what to do to change output encoding. Especially the section "StreamWriter Wrapper around Stdout" is interesting. Essentially it says to change the I/O encoding function like this:

   In Python 2:

   if sys.stdout.encoding != 'cp850':
     sys.stdout = codecs.getwriter('cp850')(sys.stdout, 'strict')
   if sys.stderr.encoding != 'cp850':
     sys.stderr = codecs.getwriter('cp850')(sys.stderr, 'strict')
   

   In Python 3:

   if sys.stdout.encoding != 'cp850':
     sys.stdout = codecs.getwriter('cp850')(sys.stdout.buffer, 'strict')
   if sys.stderr.encoding != 'cp850':
     sys.stderr = codecs.getwriter('cp850')(sys.stderr.buffer, 'strict')
   

   If used in CGI outputting HTML you can replace 'strict' by 'xmlcharrefreplace' to get HTML encoded tags for non-printable characters.

   Feel free to modify the approaches, setting different encodings, .... Note that it still wont work to output non-specified data. So any data, input, texts must be correctly convertable into unicode:

   # -*- coding: utf-8 -*-
   import sys
   import codecs
   sys.stdout = codecs.getwriter("iso-8859-1")(sys.stdout, 'xmlcharrefreplace')
   print u"Stöcker"                # works
   print "Stöcker".decode("utf-8") # works
   print "Stöcker"                 # fails