Should we HTML-encode special characters before storing them in the database?

encoding character-encoding special-characters
Mohamad picture Mohamad · Jan 4, 2011 · Viewed 19.2k times · Source

I use MySQL to store data and my web pages are all encoded as UTF-8. I have a lot of Portuguese characters such as ç and õ and I'm wondering if I should HTML-escape them before storage.

Should we store & as &, for example? And why (not)? What are the advantages and disadvantages / best practices?

Answer

Ned Batchelder picture Ned Batchelder · Jan 4, 2011

Don't HTML-encode your characters before storage. You should store as pure a form of your data as possible. HTML encoding is needed because you are going to display the data on an HTML page, so do the encoding during the processing of the data to create the page. For example, suppose you decide you're also going to send the data in plain text emails. If you've HTML-encoded the data, now the HTML encoding is a barrier that you have to undo.

Choose a canonical form for your data, and store that. UTF-8 is wonderful, and your database supports it (assuming you've created all your tables properly). Just store UTF-8.

Related questions

Working with UTF-8 encoding in Python source

Consider: $ cat bla.py u = unicode('d…') s = u.encode('utf-8') print s $ python bla.py File "bla.py", line 1 SyntaxError: Non-ASCII character '\xe2' in file bla.py on line 1, but no encoding declared; see http://…

Read More
How to convert Strings to and from UTF8 byte arrays in Java

In Java, I have a String and I want to encode it as a byte array (in UTF8, or some other encoding). Alternately, I have a byte array (in some known encoding) and I want to convert it into a …

Read More
How can I transform string to UTF-8 in C#?

I have a string that I receive from a third party app and I would like to display it correctly in any language using C# on my Windows Surface. Due to incorrect encoding, a piece of my string looks like …

Read More