Storing a string as UTF8 in C#

c# .net string utf-8 ascii
PhantomDrummer picture PhantomDrummer · Aug 27, 2012 · Viewed 27.9k times · Source

I'm doing a lot of string manipulation in C#, and really need the strings to be stored one byte per character. This is because I need gigabytes of text simultaneously in memory and it's causing low memory issues. I know for certain that this text will never contain non-ASCII characters, so for my purposes, the fact that System.String and System.Char store everything as two bytes per character is both unnecessary and a real problem.

I'm about to start coding my own CharAscii and StringAscii classes - the string one will basically hold its data as byte[], and expose string manipulation methods similar to the ones that System.String does. However this seems a lot of work to do something that seems like a very standard problem, so I'm really posting here to check that there isn't already an easier solution. Is there for example some way I can make System.String internally store data as UTF8 that I haven't noticed, or some other way round the problem?

Answer

KeithS picture KeithS · Aug 27, 2012

Well, you could create a wrapper that retrieves the data as UTF-8 bytes and converts pieces as needed to System.String, then vice-versa to push the string back out to memory. The Encoding class will help you out here:

var utf8 = Encoding.UTF8;
byte[] utfBytes = utf8.GetBytes(myString);

var myReturnedString = utf8.GetString(utfBytes);

Related questions

Why does .net use the UTF16 encoding for string, but uses UTF-8 as default for saving files?

From here Essentially, string uses the UTF-16 character encoding form But when saving vs StreamWriter : This constructor creates a StreamWriter with UTF-8 encoding without a Byte-Order Mark (BOM), I've seen this sample (broken link removed): And it looks like utf8 …

Read More
How can I convert String to Int?

I have a TextBoxD1.Text and I want to convert it to an int to store it in a database. How can I do this?

Read More
How do I get a consistent byte representation of strings in C# without manually specifying an encoding?

How do I convert a string to a byte[] in .NET (C#) without manually specifying a specific encoding? I'm going to encrypt the string. I can encrypt it without converting, but I'd still like to know why encoding comes to …

Read More