wchar_t and encoding

c++ character-encoding wchar-t
Hunter picture Hunter · May 3, 2012 · Viewed 7.1k times · Source

If I want to convert a piece of string to UTF-16, say char * xmlbuffer, do I have to convert the type to wchar_t * before encoding to UTF-16? And is char* type reqired before encoding to UTF-8?

How is wchar_t, char related to UTF-8 or UTF-16 or UTF-32 or other transformation format?

Thanks in advance for help!

Answer

Jon picture Jon · May 4, 2012

No, you don't have to change data types.

About wchar_t: the standard says that

Type wchar_t is a distinct type whose values can represent distinct codes for all members of the largest extended character set specified among the supported locales.

Unfortunately, it does not say what encoding wchar_t is supposed to have; this is implementation-dependent. So for example given 

auto s = L"foo";

you can make absolutely no assumption about what the value of the expression *s is.

However, you can use an std::string as an opaque sequence of bytes that represent text in any transformation format of your choice without issue. Just don't perform standard library string-related operations on it.

Related questions

How do you properly use WideCharToMultiByte

I've read the documentation on WideCharToMultiByte, but I'm stuck on this parameter: lpMultiByteStr [out] Pointer to a buffer that receives the converted string. I'm not quite sure how to properly initialize the variable and feed it into the function

Read More
UTF8 to/from wide char conversion in STL

Is it possible to convert UTF8 string in a std::string to std::wstring and vice versa in a platform independent manner? In a Windows application I would use MultiByteToWideChar and WideCharToMultiByte. However, the code is compiled for multiple OSes …

Read More
Using Unicode in C++ source code

What is the standard encoding of C++ source code? Does the C++ standard even say something about this? Can I write C++ source in Unicode? For example, can I use non-ASCII characters such as Chinese characters in comments? If so, …

Read More