What are the main references to the fish-eye camera model in OpenCV3.0.0dev?

Question

I am wrestling with the Fish-Eye Camera Model used in OpenCV 3.0.0.dev. I have read the documentation in this link several times, especially the "Detailed Description" part and formulas modeling fish-eye distortion. By now I have two concerns:

1. Based on the projection models listed here and their conceptual explanations in "Accuracy of Fish-Eye Lens Model" By Hughes, I can't figure out which projection model has been used in the OpenCV implementation.

2. Since the description is so concise, I need to know the main reference papers used by OpenCV developers for implementing fish-eye namespace, so that I could be on the ball and get through more details. P.S. I checked OpenCV 3.0.0-dev documentation and did not find anything useful.

Answer 1

Short answer:

OpenCV 3.0.0 Fisheye camera model doesn't use the Brown model nor any of the models that the OP refers from Panotools, it uses a Generic Camera Model by Juho Kannala and Sami S. Brandt.

Detailed answer:

As Cfr points in his answer, this comment from Ilya Krylov (who implemented the fisheye model in OpenCV) says they ported the Camera Calibration Toolbox for Matlab of Jean-Yves Bouguet:

Jean-Yves Bouguet website (link), in turn, mentions the paper A Generic Camera Model and Calibration Method for Conventional, Wide-Angle, and Fish-Eye Lenses, and says :

The "undocumented" fisheye model contained in the calibration toolbox follows the equidistance projection model described by equation (3) in this very nice paper. The distortion model follows equation (6), to the exception that k1=1 (otherwise indistinguishable from f).

Which in my opinion is a misleading statement or a plain misconception as equation(3) and equation(6) correspond to different models: equation (6) is the actual model introduced in this paper which the authors refer as the Generic Camera Model (hence the name of the paper). To be more precise, equation (6) was meant to be used as the camera model and equation (8) and (9) as the "distortion" or deviation from this model.

But the odyssey is not over. OpenCV's implementation (according to its documentation) first computes the pinhole projection to find the field angle (the angle between the 3D point, the center of projection and the optical axis). This means you can't use their fisheye model to project rays at 90º (or you would divide by 0) or close to 90º (numerical stability problems, like overflow could happen if z is small enough). Moreover I'm not sure whether it will work for rays for more than 90º. All this makes me really wonder the "usefulness" of their fisheye camera model for fisheye or wide angle lenses.

In case you're skeptic about that you can take a look at OpenCV's source code, concretely at sources\modules\calib3d\src\fisheye.cpp (I added some comments)

void cv::fisheye::projectPoints(InputArray objectPoints, OutputArray imagePoints, InputArray _rvec,
InputArray _tvec, InputArray _K, InputArray _D, double alpha, OutputArray jacobian)
{
    ...
    Rodrigues(om, R, dRdom);
    Affine3d aff(om, T);
    ...
    Vec3d Xi = objectPoints.depth() == CV_32F ? (Vec3d)Xf[i] : Xd[i];
    Vec3d Y = aff*Xi; /* To transform to camera reference frame*/

    Vec2d x(Y[0]/Y[2], Y[1]/Y[2]); /* <- The root of all evil (division by z) */

    double r2 = x.dot(x);
    double r = std::sqrt(r2);

    // Angle of the incoming ray:
    double theta = atan(r);

    double theta2 = theta*theta, theta3 = theta2*theta, theta4 = theta2*theta2, theta5 = theta4*theta,
            theta6 = theta3*theta3, theta7 = theta6*theta, theta8 = theta4*theta4, theta9 = theta8*theta;

    double theta_d = theta + k[0]*theta3 + k[1]*theta5 + k[2]*theta7 + k[3]*theta9;

    double inv_r = r > 1e-8 ? 1.0/r : 1;
    double cdist = r > 1e-8 ? theta_d * inv_r : 1;

    Vec2d xd1 = x * cdist;
    Vec2d xd3(xd1[0] + alpha*xd1[1], xd1[1]);
    Vec2d final_point(xd3[0] * f[0] + c[0], xd3[1] * f[1] + c[1]);
    ...
}

Update: This pull request fixes the problem with rays at angles ≥ 90º. As of April 2018 it hasn't been merged into master yet but is being considered for OpenCV 4.x Calibration Module (check calibration module discussion too)