How to detect eye blink using Google vision API in android ?

Question

i'm using the vision API for face detection, now i want to implement eye blink but still vision api detect eye blinking in image(photo) of a person(not live).

In addition, I am using a Tracker to keep track of the eye state over time, to detect the sequence of events that indicate a blink of left eye:

left eyes open -> left eyes closed -> left eyes open

The GraphicFaceTracker class is defined as below :

private class GraphicFaceTracker extends Tracker<Face> {
        private GraphicOverlay mOverlay;
        private FaceGraphic mFaceGraphic;
        private Context context ;

        GraphicFaceTracker(Context context, GraphicOverlay overlay) {
            mOverlay = overlay;
            this.context= context;
            mFaceGraphic = new FaceGraphic(overlay);
        }

        private final float OPEN_THRESHOLD = 0.85f;
        private final float CLOSE_THRESHOLD = 0.4f;

        private int state = 0;


        void blink(float value, final int eyeNo, String whichEye) {
            switch (state) {
                case 0:
                    if (value > OPEN_THRESHOLD) {
                        // Both eyes are initially open
                        state = 1;
                    }
                    break;

                case 1:
                    if (value < CLOSE_THRESHOLD ) {
                        // Both eyes become closed
                        state = 2;
                    }
                    break;

                case 2:
                    if (value > OPEN_THRESHOLD)  {
                        // Both eyes are open again
                        Log.i("BlinkTracker", "blink occurred!");

                        mCameraSource.takePicture(null, new CameraSource.PictureCallback() {
                            @Override
                            public void onPictureTaken(byte[] bytes) {
                                Bitmap bmp = BitmapFactory.decodeByteArray(bytes, 0, bytes.length);
                                Log.d("BITMAP", bmp.getWidth() + "x" + bmp.getHeight());
                                System.out.println(bmp.getWidth() + "x" + bmp.getHeight());
                            }
                        });
                        state = 0;
                    }
                    break;
            }


        }

        /**
         * Start tracking the detected face instance within the face overlay.
         */
        @Override
        public void onNewItem(int faceId, Face item) {
            mFaceGraphic.setId(faceId);
        }

        /**
         * Update the position/characteristics of the face within the overlay.
         */
        @Override
        public void onUpdate(FaceDetector.Detections<Face> detectionResults, Face face) {
            mOverlay.add(mFaceGraphic);
            mFaceGraphic.updateFace(face);

            float left = face.getIsLeftEyeOpenProbability();
            float right = face.getIsRightEyeOpenProbability();
            if (left == Face.UNCOMPUTED_PROBABILITY)  {
                // At least one of the eyes was not detected.
                return;
            }
            blink(left,0,"left");

            if(right == Face.UNCOMPUTED_PROBABILITY ){
                return ;
            }
        }
}

I have enabled "classifications" in order to have the detector indicate if eyes are open/closed :

FaceDetector detector = new FaceDetector.Builder(context)
            .setProminentFaceOnly(true) // optimize for single, relatively large face
            .setTrackingEnabled(true) // enable face tracking
            .setClassificationType(/* eyes open and smile */ FaceDetector.ALL_CLASSIFICATIONS)
            .setMode(FaceDetector.FAST_MODE) // for one face this is OK
            .build();

The tracker is then added as a processor for receiving face updates over time from the detector. For example, this configuration would be used to track whether the largest face in view has blinked:

Tracker<Face> tracker = new GraphicFaceTracker(this,mGraphicOverlay);
detector.setProcessor(new LargestFaceFocusingProcessor.Builder(detector, tracker).build());

But the above code detects blink in image of a person . But the image of a person cannot blink . How can I detect blink by camera ?

Answer 1

From Face object you can get below probability.

 float leftOpenScore = face.getIsLeftEyeOpenProbability();
if (leftOpenScore == Face.UNCOMPUTED_PROBABILITY) {//left eye is open }else{//left eye closed }

 float leftOpenScore = face.getIsRightEyeOpenProbability();
if (leftOpenScore == Face.UNCOMPUTED_PROBABILITY) {//Right eye is open }else{//Right eye closed }

Now you can pass this value where you want to use.