I am using OpenCV 2.4.2 on Linux. I am writing in C++. I want to track simple objects (e.g. black rectangle on the white background). Firstly I am using goodFeaturesToTrack and then calcOpticalFlowPyrLK to find those points on another image. The problem is that calcOpticalFlowPyrLK doesn't find those points.
I have found code that does it in C, which does not work in my case: http://dasl.mem.drexel.edu/~noahKuntz/openCVTut9.html
I have converted it into C++:
int main(int, char**) {
Mat imgAgray = imread("ImageA.png", CV_LOAD_IMAGE_GRAYSCALE);
Mat imgBgray = imread("ImageB.png", CV_LOAD_IMAGE_GRAYSCALE);
Mat imgC = imread("ImageC.png", CV_LOAD_IMAGE_UNCHANGED);
vector<Point2f> cornersA;
goodFeaturesToTrack(imgAgray, cornersA, 30, 0.01, 30);
for (unsigned int i = 0; i < cornersA.size(); i++) {
drawPixel(cornersA[i], &imgC, 2, blue);
}
// I have no idea what does it do
// cornerSubPix(imgAgray, cornersA, Size(15, 15), Size(-1, -1),
// TermCriteria(TermCriteria::COUNT + TermCriteria::EPS, 20, 0.03));
vector<Point2f> cornersB;
vector<uchar> status;
vector<float> error;
// winsize has to be 11 or 13, otherwise nothing is found
int winsize = 11;
int maxlvl = 5;
calcOpticalFlowPyrLK(imgAgray, imgBgray, cornersA, cornersB, status, error,
Size(winsize, winsize), maxlvl);
for (unsigned int i = 0; i < cornersB.size(); i++) {
if (status[i] == 0 || error[i] > 0) {
drawPixel(cornersB[i], &imgC, 2, red);
continue;
}
drawPixel(cornersB[i], &imgC, 2, green);
line(imgC, cornersA[i], cornersB[i], Scalar(255, 0, 0));
}
namedWindow("window", 1);
moveWindow("window", 50, 50);
imshow("window", imgC);
cvWaitKey(0);
return 0;
}
ImageA: http://oi50.tinypic.com/14kv05v.jpg
I have found out that it works only for winsize = 11. I have tried using it on a moving rectangle to check how far it is from the origin. It hardly ever detects all four corners.
int main(int, char**) {
std::cout << "Compiled at " << __TIME__ << std::endl;
Scalar white = Scalar(255, 255, 255);
Scalar black = Scalar(0, 0, 0);
Scalar red = Scalar(0, 0, 255);
Rect rect = Rect(50, 100, 100, 150);
Mat org = Mat(Size(640, 480), CV_8UC1, white);
rectangle(org, rect, black, -1, 0, 0);
vector<Point2f> features;
goodFeaturesToTrack(org, features, 30, 0.01, 30);
std::cout << "POINTS FOUND:" << std::endl;
for (unsigned int i = 0; i < features.size(); i++) {
std::cout << "Point found: " << features[i].x;
std::cout << " " << features[i].y << std::endl;
}
bool goRight = 1;
while (1) {
if (goRight) {
rect.x += 30;
rect.y += 30;
if (rect.x >= 250) {
goRight = 0;
}
} else {
rect.x -= 30;
rect.y -= 30;
if (rect.x <= 50) {
goRight = 1;
}
}
Mat frame = Mat(Size(640, 480), CV_8UC1, white);
rectangle(frame, rect, black, -1, 0, 0);
vector<Point2f> found;
vector<uchar> status;
vector<float> error;
calcOpticalFlowPyrLK(org, frame, features, found, status, error,
Size(11, 11), 5);
Mat display;
cvtColor(frame, display, CV_GRAY2BGR);
for (unsigned int i = 0; i < found.size(); i++) {
if (status[i] == 0 || error[i] > 0) {
continue;
} else {
line(display, features[i], found[i], red);
}
}
namedWindow("window", 1);
moveWindow("window", 50, 50);
imshow("window", display);
if (cvWaitKey(300) > 0) {
break;
}
}
}
OpenCV implementation of Lucas-Kanade seems to be unable to track a rectangle on a binary image. Am I doing something wrong or does this function just not work?
The Lucas Kanade method estimates the motion of a region by using the gradients in that region. It is in a case a gradient descends methods. So if you don't have gradients in x AND y direction the method will fail. The second important note is that the Lucas Kanade equation
E = sum_{winsize} (Ix * u + Iy * v * It)²
is an first order taylor approximation of the intensity constancy constrain.
I(x,y,t) = I(x+u,y+v,t+1)
so an restriction of the method without level (image pyramids) is that the image needs to be a linear function. In practise this mean only small motions could be estimated, dependend from the winsize you choose. Thats why you use the levels, which linearise the images (It). So a level of 5 is a little bit to high 3 should be enough. The top level image has in your case a size of 640x480 / 2^5 = 20 x 15.
Finally the problem in your code is the line:
if (status[i] == 0 || error[i] > 0) {
the error you get back from the lucas kanade method is the resulting SSD that means:
error = sum(winSize) (I(x,y,0) - I(x+u,y+u,1)^2) / (winsize * winsize)
It is very unlikely that the error is 0. So finally you skip all features. I have good experiences by ignoring the error, that is just a confidence measure. There are very good alternative confidence measures as the Foreward/Backward confidence. You could also start experiments by ignoring the status flag if too much feaurtes are discard