Auto-Detect Character Encoding in Java

java encoding io bufferedreader
Kirk picture Kirk · Feb 7, 2012 · Viewed 22.8k times · Source

Seems to be a fairly hit issue, but I've not yet been able to find a solution; perhaps because it comes in so many flavors. Here it is though. I'm trying to read some comma delimited files (occasionally the delimiters can be a little bit more unique than commas, but commas will suffice for now).

The files are supposed to be standardized across the industry, but lately we've seen many different types of character set files coming in. I'd like to be able to set up a BufferedReader to compensate for this.

What is a pretty standard way of doing this and detecting whether it was successful or not?

My first thoughts on this approach are to loop through character sets simple->complex until I can read the file without an exception. Not exactly ideal though...

Thanks for your attention.

Answer

Aravind Yarram picture Aravind Yarram · Feb 7, 2012

The Mozilla's universalchardet is supposed to be the efficient detector out there. juniversalchardet is the java port of it. There is one more port. Read this SO for more information Character Encoding Detection Algorithm

Related questions

Java URL encoding of query string parameters

Say I have a URL http://example.com/query?q= and I have a query entered by the user such as: random word £500 bank $ I want the result to be a properly encoded URL: http://example.com/query?q=random%20…

Read More
How to convert Strings to and from UTF8 byte arrays in Java

In Java, I have a String and I want to encode it as a byte array (in UTF8, or some other encoding). Alternately, I have a byte array (in some known encoding) and I want to convert it into a …

Read More
Java : How to determine the correct charset encoding of a stream

With reference to the following thread: Java App : Unable to read iso-8859-1 encoded file correctly What is the best way to programatically determine the correct charset encoding of an inputstream/file ? I have tried using the following: File in = …

Read More