Convert Android camera2 api YUV_420_888 to RGB

Question

I am writing an app that takes the camera feed, converts it to rgb, in order to do some processing.

It works fine on the old camera implementation which uses NV21 Yuv format. The issue I am having is with the new Yuv format, YUV_420_888. The image is no longer converted correctly to RGB in the new Camera2 Api which sends YUV_420_888 yuv format instead of NV21 (YUV_420_SP) format.

Can someone please tell me how should I convert YUV_420_888 to RGB?

Thanks

Answer 1

In my approach I use OpenCV Mat and script from https://gist.github.com/camdenfullmer/dfd83dfb0973663a7974

First of all you convert your YUV_420_888 Image to Mat with the code in the link above.

*mImage is my Image object which i get in ImageReader.OnImageAvailableListener

Mat mYuvMat = imageToMat(mImage);

public static Mat imageToMat(Image image) {
    ByteBuffer buffer;
    int rowStride;
    int pixelStride;
    int width = image.getWidth();
    int height = image.getHeight();
    int offset = 0;

    Image.Plane[] planes = image.getPlanes();
    byte[] data = new byte[image.getWidth() * image.getHeight() * ImageFormat.getBitsPerPixel(ImageFormat.YUV_420_888) / 8];
    byte[] rowData = new byte[planes[0].getRowStride()];

    for (int i = 0; i < planes.length; i++) {
        buffer = planes[i].getBuffer();
        rowStride = planes[i].getRowStride();
        pixelStride = planes[i].getPixelStride();
        int w = (i == 0) ? width : width / 2;
        int h = (i == 0) ? height : height / 2;
        for (int row = 0; row < h; row++) {
            int bytesPerPixel = ImageFormat.getBitsPerPixel(ImageFormat.YUV_420_888) / 8;
            if (pixelStride == bytesPerPixel) {
                int length = w * bytesPerPixel;
                buffer.get(data, offset, length);

                if (h - row != 1) {
                    buffer.position(buffer.position() + rowStride - length);
                }
                offset += length;
            } else {


                if (h - row == 1) {
                    buffer.get(rowData, 0, width - pixelStride + 1);
                } else {
                    buffer.get(rowData, 0, rowStride);
                }

                for (int col = 0; col < w; col++) {
                    data[offset++] = rowData[col * pixelStride];
                }
            }
        }
    }

    Mat mat = new Mat(height + height / 2, width, CvType.CV_8UC1);
    mat.put(0, 0, data);

    return mat;
}

We have 1 channel YUV Mat. Define new Mat for BGR(not RGB yet) image:

Mat bgrMat = new Mat(mImage.getHeight(), mImage.getWidth(),CvType.CV_8UC4);

I just started learning OpenCV so propably this doesn't have to be 4-channel Mat and instead could be 3-channel but it works for me. Now I use convert color method to change my yuv Mat into bgr Mat.

Imgproc.cvtColor(mYuvMat, bgrMat, Imgproc.COLOR_YUV2BGR_I420);

Now we can do all the image processing like finding contours, colors, circles, etc. To print image back on screen we need to convert it to bitmap:

Mat rgbaMatOut = new Mat();
Imgproc.cvtColor(bgrMat, rgbaMatOut, Imgproc.COLOR_BGR2RGBA, 0);
final Bitmap bitmap = Bitmap.createBitmap(bgrMat.cols(), bgrMat.rows(), Bitmap.Config.ARGB_8888);
Utils.matToBitmap(rgbaMatOut, bitmap);

I have all my image processing in seperate thread so to set my ImageView I need to do this on the UI thread.

runOnUiThread(new Runnable() {
                    @Override
                    public void run() {
                        if(bitmap != null) {
                            mImageView.setImageBitmap(bitmap);
                        }
                    }
                });