Convert zero-padded bytes to UTF-8 string

python unicode utf-8 byte strncpy
Matt Joiner picture Matt Joiner · Feb 22, 2011 · Viewed 26.9k times · Source

I'm unpacking several structs that contain 's' type fields from C. The fields contain zero-padded UTF-8 strings handled by strncpy in the C code (note this function's vestigial behaviour). If I decode the bytes I get a unicode string with lots of NUL characters on the end.

>>> b'hiya\0\0\0'.decode('utf8')
'hiya\x00\x00\x00'

I was under the impression that trailing zero bytes were part of UTF-8 and would be dropped automatically.

What's the proper way to drop the zero bytes?

Answer

Adam Rosenfield picture Adam Rosenfield · Feb 22, 2011

Use str.rstrip() to remove the trailing NULs:

>>> 'hiya\0\0\0'.rstrip('\0')
'hiya'

Related questions

Unicode (UTF-8) reading and writing to files in Python

I'm having some brain failure in understanding reading and writing text to a file (Python 2.4). # The string, which has an a-acute in it. ss = u'Capit\xe1n' ss8 = ss.encode('utf8') repr(ss), repr(ss8) ("u'Capit\xe1n'", "'Capit\…

Read More
How to convert a string to utf-8 in Python

I have a browser which sends utf-8 characters to my Python server, but when I retrieve it from the query string, the encoding that Python returns is ASCII. How can I convert the plain string to utf-8? NOTE: The string …

Read More
Convert Unicode to ASCII without errors in Python

My code just scrapes a web page, then converts it to Unicode. html = urllib.urlopen(link).read() html.encode("utf8","ignore") self.response.out.write(html) But I get a UnicodeDecodeError: Traceback (most recent call last): File "/Applications/GoogleAppEngineLauncher.app/…

Read More