Face Recognition on Android

Question

I'm trying to develop a Face Recognition app on Android and since I don't want to use NDK on the project (simply don't have the time to switch), I'm sticking to develop the whole app with Java and therefor I'm having some problems :

1. It seems the Contrib Module isn't included in OpenCV 2.4.2 . is there anyway to use it in the project ?

2. I tried using JavaCV to use the Contrib Module's "FaceRecognizer" class. there are two classes available called "FaceRecognizer" & "FaceRecognizerPtr". does anybody know what the difference between these two is ?

3. The classes mentioned above have a method called "Train" which (In C++) receives two Vectors of types "Mat & Integer" ( model->train(images,labels) & train(Vector<mat> theImages, Vector<int> theLabels) . I tried passing them ArrayList<mat> & ArrayList<integer> and Vectors in Java but it seems that the method explicitly accepts the "CvArr" Data type which I'm not sure how to acquire... Here is the error :

The method train(opencv_core.CvArr, opencv_core.CvArr) in the type opencv_contrib.FaceRecognizer is not applicable for the arguments (ArrayList, ArrayList)

Does anyone know how to change my ArrayList to CvArr ?!

This is my first post and I wasn't sure whether to ask all three questions in one post or in three posts so sorry for any inconveniences... If you need any other Information about the project, feel free to ask.

Answer 1

Update

The following article was written by Petter Christian Bjelland, so all credit is his. I am posting it here, because his blog seems to be in Maintenance mode at the moment, but I think it is worth sharing.

Doing face recognition with JavaCV (from http://pcbje.com)

I couldn’t find any tutorial on how to perform face recognition using OpenCV and Java, so I decided to share a viable solution here. The solution is very inefficient in its current form as the training model is built at each run, however it shows what’s needed to make it work.

The class below takes two arguments: The path to the directory containing the training faces and the path to the image you want to classify. Not that all images has to be of the same size and that the faces already has to be cropped out of their original images (Take a look here if you haven’t done the face detection yet).

For the simplicity of this post, the class also requires that the training images have filename format: <label>-rest_of_filename.png. For example:

1-jon_doe_1.png
1-jon_doe_2.png
2-jane_doe_1.png
2-jane_doe_2.png

... and so on.

The code:

import com.googlecode.javacv.cpp.opencv_core;
import static com.googlecode.javacv.cpp.opencv_highgui.*;
import static com.googlecode.javacv.cpp.opencv_core.*;
import static com.googlecode.javacv.cpp.opencv_imgproc.*;
import static com.googlecode.javacv.cpp.opencv_contrib.*;
import java.io.File;
import java.io.FilenameFilter;

public class OpenCVFaceRecognizer {
  public static void main(String[] args) {
    String trainingDir = args[0];
    IplImage testImage = cvLoadImage(args[1]);

    File root = new File(trainingDir);

    FilenameFilter pngFilter = new FilenameFilter() {
      public boolean accept(File dir, String name) {
        return name.toLowerCase().endsWith(".png");
      }
    };

    File[] imageFiles = root.listFiles(pngFilter);

    MatVector images = new MatVector(imageFiles.length);

    int[] labels = new int[imageFiles.length];

    int counter = 0;
    int label;

    IplImage img;
    IplImage grayImg;

    for (File image : imageFiles) {
      // Get image and label:
      img = cvLoadImage(image.getAbsolutePath());
      label = Integer.parseInt(image.getName().split("\\-")[0]);
      // Convert image to grayscale:
      grayImg = IplImage.create(img.width(), img.height(), IPL_DEPTH_8U, 1);
      cvCvtColor(img, grayImg, CV_BGR2GRAY);
      // Append it in the image list:
      images.put(counter, grayImg);
      // And in the labels list:
      labels[counter] = label;
      // Increase counter for next image:
      counter++;
    }

    FaceRecognizer faceRecognizer = createFisherFaceRecognizer();
    // FaceRecognizer faceRecognizer = createEigenFaceRecognizer();
    // FaceRecognizer faceRecognizer = createLBPHFaceRecognizer()

    faceRecognizer.train(images, labels);

    // Load the test image:
    IplImage greyTestImage = IplImage.create(testImage.width(), testImage.height(), IPL_DEPTH_8U, 1);
    cvCvtColor(testImage, greyTestImage, CV_BGR2GRAY);

    // And get a prediction:
    int predictedLabel = faceRecognizer.predict(greyTestImage);
    System.out.println("Predicted label: " + predictedLabel);
  }
}

The class requires the OpenCV Java interface. If you’re using Maven, you can retrieve the required libraries with the following pom.xml:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
     xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <groupId>com.pcbje</groupId>
  <artifactId>opencvfacerecognizer</artifactId>
  <version>0.1-SNAPSHOT</version>
  <packaging>jar</packaging>

  <name>opencvfacerecognizer</name>
  <url>http://pcbje.com</url>

  <dependencies>
    <dependency>
      <groupId>com.googlecode.javacv</groupId>
      <artifactId>javacv</artifactId>
      <version>0.3</version>
    </dependency>

    <!-- For Linux x64 environments -->
    <dependency>
      <groupId>com.googlecode.javacv</groupId>
      <artifactId>javacv</artifactId>
      <classifier>linux-x86_64</classifier>
      <version>0.3</version>
    </dependency>    

    <!-- For OSX environments -->
    <dependency>
      <groupId>com.googlecode.javacv</groupId>
      <artifactId>javacv</artifactId>
      <classifier>macosx-x86_64</classifier>
      <version>0.3</version>
    </dependency>
  </dependencies>

  <repositories>
    <repository>
      <id>javacv</id>
      <name>JavaCV</name>
      <url>http://maven2.javacv.googlecode.com/git/</url>
    </repository>
  </repositories>
</project>

Original Post

Quoting from my reply on http://answers.opencv.org/question/865/the-contrib-module-problem.

Without ever having used javacv, let's see how far we can get by just looking at the interfaces! The project is on googlecode, which makes it easy to browse the code: http://code.google.com/p/javacv.

First have a look at how cv::FaceRecognizer has been wrapped (opencv_contrib.java, line 845 at time of writing this):

@Namespace("cv") public static class FaceRecognizer extends Algorithm {
    static { Loader.load(); }
    public FaceRecognizer() { }
    public FaceRecognizer(Pointer p) { super(p); }

    public /*abstract*/ native void train(@ByRef MatVector src, @Adapter("ArrayAdapter") CvArr labels);
    public /*abstract*/ native int predict(@Adapter("ArrayAdapter") CvArr src);
    public /*abstract*/ native void predict(@Adapter("ArrayAdapter") CvArr src, @ByRef int[] label, @ByRef double[] dist);
    public native void save(String filename);
    public native void load(String filename);
    public native void save(@Adapter("FileStorageAdapter") CvFileStorage fs);
    public native void load(@Adapter("FileStorageAdapter") CvFileStorage fs);
}

Aha, so you need to pass a MatVector for the images! You can pass the labels in a CvArr (one row or one column). The MatVector is defined in opencv_core, line 4629 (at time of writing this) and it looks like this:

public static class MatVector extends Pointer {
    static { load(); }
    public MatVector()       { allocate();  }
    public MatVector(long n) { allocate(n); }
    public MatVector(Pointer p) { super(p); }
    private native void allocate();
    private native void allocate(@Cast("size_t") long n);

    public native long size();
    public native void resize(@Cast("size_t") long n);

    @Index @ValueGetter public native @Adapter("MatAdapter") CvMat getCvMat(@Cast("size_t") long i);
    @Index @ValueGetter public native @Adapter("MatAdapter") CvMatND getCvMatND(@Cast("size_t") long i);
    @Index @ValueGetter public native @Adapter("MatAdapter") IplImage getIplImage(@Cast("size_t") long i);
    @Index @ValueSetter public native MatVector put(@Cast("size_t") long i, @Adapter("MatAdapter") CvArr value);
}

Again just by looking at the code, I guess it can be used like this:

int numberOfImages = 10;
// Allocate some memory:
MatVector images = new MatVector(numberOfImages);
// Then fill the MatVector, you probably want to do something useful instead:
for(int idx = 0; idx < numberOfImages; idx++){
   // Load an image:
   CvArr image = cvLoadImage("/path/to/your/image");
   // And put it into the MatVector:
   images.put(idx, image);
}

You probably want to write yourself a method that does the conversion from a Java ArrayList to a MatVector (if such a function does not exist in javacv yet).

Now to your second question. FaceRecognizer is the equivalent to cv::FaceRecognizer. The native OpenCV C++ classes return a cv::Ptr<cv::FaceRecognizer>, which is a (Smart) Pointer to a cv::FaceRecognizer. This has to be wrapped as well. See a pattern here?

The interface of FaceRecognizerPtr now looks like this:

@Name("cv::Ptr<cv::FaceRecognizer>")
public static class FaceRecognizerPtr extends Pointer {
    static { load(); }
    public FaceRecognizerPtr()       { allocate();  }
    public FaceRecognizerPtr(Pointer p) { super(p); }
    private native void allocate();

    public native FaceRecognizer get();
    public native FaceRecognizerPtr put(FaceRecognizer value);
}

So you can either get a FaceRecognizer from this class or put a FaceRecognizer into. You should only be concerned about the get(), as the Pointer is filled by the method creating the concrete FaceRecognizer algorithm:

@Namespace("cv") public static native @ByVal FaceRecognizerPtr createEigenFaceRecognizer(int num_components/*=0*/, double threshold/*=DBL_MAX*/);
@Namespace("cv") public static native @ByVal FaceRecognizerPtr createFisherFaceRecognizer(int num_components/*=0*/, double threshold/*=DBL_MAX*/);
@Namespace("cv") public static native @ByVal FaceRecognizerPtr createLBPHFaceRecognizer(int radius/*=1*/,
        int neighbors/*=8*/, int grid_x/*=8*/, int grid_y/*=8*/, double threshold/*=DBL_MAX*/);

So once you have got the FaceRecognizerPtr, you can do things like:

// Holds your training data and labels:
MatVector images;
CvArr labels;
// Do something with the images and labels... Probably fill them?
// ...
// Then get a Pointer to a FaceRecognizer (FaceRecognizerPtr).
// Java doesn't have default parameters, so you have to add some yourself,
// if you pass 0 as num_components to the EigenFaceRecognizer, the number of
// components is determined by the data, for the threshold use the maximum possible
// value if you don't want one. I don't know the constant in Java:
FaceRecognizerPtr model = createEigenFaceRecognizer(0, 10000);
// Then train it. See how I call get(), to get the FaceRecognizer inside the FaceRecognizerPtr:
model.get().train(images, labels);

This learns you an Eigenfaces model. And that's it!