Mobile Vision API - concatenate new detector object to continue frame processing

Question

I want to use the new face detection feature that the vision API provides along with additional frame processing in an application. For this, I need to have access to the camera frame that was processed by the face detector, and concatenate a processor using the face detected data.

As I see in the sample, the CameraSource abstracts the detection and camera access, and I can't have access to the frame being processed. Are there examples of how to get the camera frame in this API, or, maybe, create and concatenate a detector that receives it? Is that possible at least?

Thanks, Lucio

Answer 1

Yes, it is possible. You'd need to create your own subclass of Detector which wraps FaceDetector and executes your extra frame processing code in the detect method. It would look something like this:

class MyFaceDetector extends Detector<Face> {
  private Detector<Face> mDelegate;

  MyFaceDetector(Detector<Face> delegate) {
    mDelegate = delegate;
  }

  public SparseArray<Face> detect(Frame frame) {
    // *** add your custom frame processing code here
    return mDelegate.detect(frame);
  }

  public boolean isOperational() {
    return mDelegate.isOperational();
  }

  public boolean setFocus(int id) {
    return mDelegate.setFocus(id);
  }
}

You'd wrap the face detector with your class, and pass your class into the camera source. It would look something like this:

    FaceDetector faceDetector = new FaceDetector.Builder(context)
            .build();
    MyFaceDetector myFaceDetector = new MyFaceDetector(faceDetector);

    myFaceDetector.setProcessor(/* include your processor here */);

    mCameraSource = new CameraSource.Builder(context, myFaceDetector)
            .build();

Your detector will be called first with the raw frame data.

Note that the image may not be upright, if the device is rotated. You can get the orientation through the frame's metadata.getRotation method.

One word of caution: once the detect method returns, you should not access the frame pixel data. Since the camera source recycles image buffers, the contents of the frame object will be eventually overridden once the method returns.

EDIT: (additional notes) You could also avoid the boilerplate code of MyFaceDetector using a MultiDetector like this:

MultiDetector multiDetector = new MultiDetector.Builder()
    .add(new FaceDetector.Builder(context)
                .build())
    .add(new YourReallyOwnDetector())
    .build();

Also note the use of FaceTrackerFactory in conjuction with MultiProcessor described there.