I'm trying to perform a complex warp of an image using Dense Optical Flow. I am trying to warp the second image into roughly the same shape as the first image.
cv::Mat flow;
cv::calcOpticalFlowFarneback( mGrayFrame1, mGrayFrame2, flow, 0.5, 3, 15, 3, 5, 1.2, 0 );
cv::Mat newFrame = cv::Mat::zeros( frame.rows, frame.cols, frame.type() );
cv:remap( frame, newFrame, flow, cv::Mat(), CV_INTER_LINEAR );
I calculate the flow from two grayscale frames. I am now trying to remap my original (i.e. non-grayscale) image using this flow information using the cv::remap
function. However, I get a very badly distorted image from it. I simply end up with an orange and black image that bears a small resemblance to my original image.
How do I use cv::remap
with the calculated flow
?
The remap
function cannot work with flow
directly. One must use a separate map
that is computed by taking the backwards flow (from frame2
to frame1
) and then offsetting each flow vector by its (x, y)
location on the pixel grid. See details below.
Recall the backwards optical flow formula:
frame1(x, y) = frame2(x + flowx(x, y), y + flowy(x, y))
The remap
function transforms the source image using a specified map
:
dst(x, y) = src(mapx(x, y), mapy(x, y))
Comparing the two equations above, we may determine the map
that remap
requires:
mapx(x, y) = x + flowx(x, y)
mapy(x, y) = y + flowy(x, y)
Example:
Mat flow; // backward flow
calcOpticalFlowFarneback(nextFrame, prevFrame, flow);
Mat map(flow.size(), CV_32FC2);
for (int y = 0; y < map.rows; ++y)
{
for (int x = 0; x < map.cols; ++x)
{
Point2f f = flow.at<Point2f>(y, x);
map.at<Point2f>(y, x) = Point2f(x + f.x, y + f.y);
}
}
Mat newFrame;
remap(prevFrame, newFrame, map);