UTF-8 characters missing or displayed as boxes in Notepad, but works fine in webbrowser and other text editors

windows utf-8 character-encoding notepad
JAVAGeek picture JAVAGeek · Jul 10, 2012 · Viewed 16.8k times · Source

I have UTF-8 text stored in DB and served as text/plain; charset=utf-8 in a web application. All the things are working fine. I can see the UTF-8 text on browser window without any problem.

But when I save that text to a file and try to open it in Windows Notepad, I got some characters missing and displayed as a small rectangular box. However, the text file looks fine in other editors like EditPlus and Notepad++.

How is this caused and how can I solve it?

Answer

Sergei Tachenov picture Sergei Tachenov · Jul 10, 2012

If it looks fine in other editors, then the text itself is fine. If it looks OK in the browser, then the response is probably fine too (but better check page info in the browser and see what the encoding is). Your problem is probably with notepad itself. Sometimes it requires BOM to detect Unicode properly. But BOM can break other apps that don't support it. You should also try Notepad on different versions of Windows. I have just tried opening an UTF-8 file in Windows 7, looks fine to me.

Related questions

Unicode (utf-8) with git-bash

I'm having some trouble getting unicode to work for git-bash (on windows 7). I have tried many things without success. Although, I'm not quite sure what is responsible to for this so i might be working in the wrong direction. It …

Read More
UTF-16 to UTF-8 conversion (for scripting in Windows)

what is the best way to convert a UTF-16 files to UTF-8? I need to use this in a cmd script.

Read More
Storing unicode UTF-8 string in std::string

In response to discussion in Cross-platform strings (and Unicode) in C++ How to deal with Unicode strings in C/C++ in a cross-platform friendly way? I'm trying to assign a UTF-8 string to a std::string variable in Visual Studio 2010 …

Read More