OpenCV Android Background Subtraction

Question

I am working on a robotics project using an Android phone as the main processor and the camera to detect movement. I got the Android binary package from OpenCV and got it correctly installed. I can capture images using the OpenCV native camera and display them to the screen. I'm having problems using the background subtraction class, though. I can make a new BackgroundSubtractorMOG object in the constructor, but when I attempt to run the code below, it force quits I get the error "Only 1- and 3-channel 8-bit images are supported in BackgroundSubtractorMOG" from the native code. I tried changing Highgui.CV_CAP_ANDROID_COLOR_FRAME_RGBA to Highgui.CV_CAP_ANDROID_COLOR_FRAME_RGB, and then it doesn't force quit, but all I get is a black screen. I'm pretty sure bmp is still null with FRAME_RGB, because the screen stays black, and the fps counter I was drawing right after the bitmap (removed from the code posted below for clarity and as a troubleshooting step) doesn't show up.

I took a look at the OpenCV C++ code for this function (line 388 here), and the image type error occurs if the image type isn't CV_8UC1 or CV_8UC3, so I tried using the java CvType.CV_8UC3 instead of Highgui.CV_CAP_ANDROID_COLOR_FRAME_RGBA in capture.retrieve(), but it force closed and I got a "Output frame format is not supported" error.

I'm guessing I've just got a type conversion problem, but I can't figure out for the life of me where OpenCV's Android-specific image types fit with their regular image types that are documented. Any help would be appreciated.

The variables:

private SurfaceHolder mHolder;
private VideoCapture mCamera;
private Mat mRgba;
private Mat mFGMask;
private BackgroundSubtractorMOG mBGSub;

My SurfaceView's run() function:

public void run() {    
    Bitmap bmp = null;

    synchronized (this) {
        if (mCamera == null)
            break;

        if (!mCamera.grab()) {
            Log.e(TAG, "mCamera.grab() failed");
            break;
        }

        processFrame(mCamera);
        bmp = Bitmap.createBitmap(mFGMask.cols(), mFGMask.rows(), Bitmap.Config.ARGB_8888);
        Utils.matToBitmap(mFGMask, bmp);
    }

    if (bmp != null) {
        Canvas canvas = mHolder.lockCanvas();
        if (canvas != null) {
            canvas.drawBitmap(bmp, (canvas.getWidth() - bmp.getWidth()) / 2, (canvas.getHeight() - bmp.getHeight()) / 2, null);
            mHolder.unlockCanvasAndPost(canvas);
        }
        bmp.recycle();
    }
}

The processFrame() function referenced in run():

protected void processFrame(VideoCapture capture) {
    capture.retrieve(mRgba, Highgui.CV_CAP_ANDROID_COLOR_FRAME_RGBA);
    mBGSub.apply(mRgba, mFGMask);
}

Edit:

The solution that ended up working:

protected void processFrame(VideoCapture capture) {
    capture.retrieve(mRgba, Highgui.CV_CAP_ANDROID_COLOR_FRAME_RGB);
    //GREY_FRAME also works and exhibits better performance
    //capture.retrieve(mRgba, Highgui.CV_CAP_ANDROID_GREY_FRAME);
    mBGSub.apply(mRgba, mFGMask, 0.1);
    Imgproc.cvtColor(mFGMask, mRgba, Imgproc.COLOR_GRAY2BGRA, 4);
}

Answer 1

Have you tried using cvtColor with CV_RGB2RGBA and CV_RGBA2RGB. So, maybe try converting frame RGBA to RGB, then do background subtraction. Something like this:

protected void processFrame(VideoCapture capture) {
    capture.retrieve(mRgba, Highgui.CV_CAP_ANDROID_COLOR_FRAME_RGBA);
    Mat rgb;
    Imgproc.cvtColor(mRgba, rgb, Imgproc.COLOR_RGBA2RGB);
    mBGSub.apply(rgb, mFGMask);
}

EDIT : You might check out the OpenCV unit-test for BackgroundSubtractorMOG located here. However, the test has fail("Not yet implemented"); in the main test case.

I'm not sure if that means the test isn't complete, or the support for BackgroundSubtractorMOG is not implemented. You might try running the code contained in this unit-test to see if it actually works.

Also, the C++ sample segment_objects.cpp might be helpful as a usage example.

Hope that helps! :)