pysqlite2: ProgrammingError - You must not use 8-bit bytestrings
I'm currently persisting filenames in a sqlite database for my own purposes. Whenever I try to insert a file that has a special character (like é etc.), it throws the following error:
pysqlite2.dbapi2.ProgrammingError: You must not use 8-bit bytestrings unless you use a text_factory that can interpret 8-bit bytestrings (like text_factory = str). It is highly recommended that you instead just switch your application to Unicode strings.
When I do "switch my application over to Unicode strings" by wrapping the value sent to pysqlite with the unicode method like: unicode(filename), it throws this error:
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 66: ordinal not in range(128)
Is there something I can do to get rid of this? Modifying all of my files to conform isn't an option.
UPDATE
If I decode the text via filename.decode("utf-8"), I'm still getting the ProgrammingError above.
My actual code looks like this:
cursor.execute("select * from musiclibrary where absolutepath = ?;",
[filename.decode("utf-8")])
What should my code here look like?
Answer
You need to specify the encoding of filename for conversion to Unicode, for example: filename.decode('utf-8'). Just using unicode(...) picks the console encoding, which is often unreliable (and often ascii).