Handling Unicode surrogate values in Java strings

Question

Consider the following code:

byte aBytes[] = { (byte)0xff,0x01,0,0,
                  (byte)0xd9,(byte)0x65,
                  (byte)0x03,(byte)0x04, (byte)0x05, (byte)0x06, (byte)0x07,
                  (byte)0x17,(byte)0x33, (byte)0x74, (byte)0x6f,
                   0, 1, 2, 3, 4, 5,
                   0 };
String sCompressedBytes = new String(aBytes, "UTF-16");
for (int i=0; i<sCompressedBytes.length; i++) {
    System.out.println(Integer.toHexString(sCompressedBytes.codePointAt(i)));
}

Gets the following incorrect output:

ff01, 0, fffd, 506, 717, 3374, 6f00, 102, 304, 500.

However, if the 0xd9 in the input data is changed to 0x9d, then the following correct output is obtained:

ff01, 0, 9d65, 304, 506, 717, 3374, 6f00, 102, 304, 500.

I realize that the functionality is because of the fact that the byte 0xd9 is a high-surrogate Unicode marker.

Question: Is there a way to feed, identify and extract surrogate bytes (0xd800 to 0xdfff) in a Java Unicode string?
Thanks

Answer 1

EDIT: This addresses the question from the comment

If you want to encode arbitrary binary data in a string, you should not use a normal text encoding. You don't have valid text in that encoding - you just have arbitrary binary data.

Base64 is the way to go here. There's no base64 support directly in Java (in a public class, anyway) but there are various 3rd party libraries you can use, such as the one in the Apache Commons Codec library.

Yes, base64 will increase the size of the data - but it'll allow you to decode it later without losing information.

EDIT: This addresses the original question

I believe that the problem is that you haven't specified a proper surrogate pair. You should specify bytes representing a low surrogate and then a high surrogate. After that, you should be able to extra the appropriate code point. In your case, you've given a low surrogate on its own.

Here's code to demonstrate this:

public class Test
{
    public static void main(String[] args)
        throws Exception // Just for simplicity
    {
        byte[] data = 
        {
            0, 0x41, // A
            (byte) 0xD8, 1, // High surrogate
            (byte) 0xDC, 2, // Low surrogate
            0, 0x42, // B
        };

        String text = new String(data, "UTF-16");

        System.out.printf("%x\r\n", text.codePointAt(0));
        System.out.printf("%x\r\n", text.codePointAt(1));
        // Code point at 2 is part of the surrogate pair
        System.out.printf("%x\r\n", text.codePointAt(3));       
    }
}

Output:

41
10402
42