Check for valid utf8 string in Python

python json utf-8 invalid-characters
troex picture troex · Mar 10, 2011 · Viewed 16.2k times · Source

I'm reading filenames from file system and I want to send them as JSON encoded array. The problem is that files on file system can be stored in invalid encoding, and I need to handle this situation to omit invalid filenames before passing it to json.dump, otherwise it will fail.

Is there a way to check that my string (filename) contains valid utf-8 chars?

Answer

Mark Longair picture Mark Longair · Mar 10, 2011

How about trying the following?

valid_utf8 = True
try:
    filename.decode('utf-8')
except UnicodeDecodeError:
    valid_utf8 = False

... based on an answer to a similar question here: How to write a check in python to see if file is valid UTF-8?

Related questions

Saving utf-8 texts in json.dumps as UTF8, not as \u escape sequence

sample code: >>> import json >>> json_string = json.dumps("ברי צקלה") >>> print json_string "\u05d1\u05e8\u05d9 \u05e6\u05e7\u05dc\u05d4" The problem: it's not human readable. …

Read More
python byte string encode and decode

I am trying to convert an incoming byte string that contains non-ascii characters into a valid utf-8 string such that I can dump is as json. b = '\x80' u8 = b.encode('utf-8') j = json.dumps(u8) I …

Read More
Why can't Python parse this JSON data?

I have this JSON in a file: { "maps": [ { "id": "blabla", "iscategorical": "0" }, { "id": "blabla", "iscategorical": "0" } ], "masks": [ "id": "valore" ], "om_points": "value", "parameters": [ "id": "valore" ] } I wrote this script to print all of the JSON data: import json from pprint import pprint …

Read More