How do I convert from a CVPixelBufferRef to an openCV cv::Mat

Question

I would like to perform a few operations to a CVPixelBufferRef and come out with a cv::Mat

• crop to a region of interest
• scaled to a fixed dimension
• equalised the histogram
• convert to greyscale - 8 bits per pixel (CV_8UC1)

I am not sure what the most efficient order is to do this, however, I do know that all of the operations are available on an open:CV matrix, so I would like to know how to convert it.

- (void) captureOutput:(AVCaptureOutput *)captureOutput 
         didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer 
         fromConnection:(AVCaptureConnection *)connection
{
     CVPixelBufferRef pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);

     cv::Mat frame = f(pixelBuffer); // how do I implement f()?

Answer 1

I found the answer in some excellent GitHub source code. I adapted it here for simplicity. It also does the greyscale conversion for me.

CVPixelBufferRef pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
OSType format = CVPixelBufferGetPixelFormatType(pixelBuffer);

// Set the following dict on AVCaptureVideoDataOutput's videoSettings to get YUV output
// @{ kCVPixelBufferPixelFormatTypeKey : kCVPixelFormatType_420YpCbCr8BiPlanarFullRange }

NSAssert(format == kCVPixelFormatType_420YpCbCr8BiPlanarFullRange, @"Only YUV is supported");

// The first plane / channel (at index 0) is the grayscale plane
// See more infomation about the YUV format
// http://en.wikipedia.org/wiki/YUV
CVPixelBufferLockBaseAddress(pixelBuffer, 0);
void *baseaddress = CVPixelBufferGetBaseAddressOfPlane(pixelBuffer, 0);

CGFloat width = CVPixelBufferGetWidth(pixelBuffer);
CGFloat height = CVPixelBufferGetHeight(pixelBuffer);

cv::Mat mat(height, width, CV_8UC1, baseaddress, 0);

// Use the mat here

CVPixelBufferUnlockBaseAddress(pixelBuffer, 0);

I am thinking that the best order will be:

1. Convert to grayscale (since it is done almost automatically)
2. Crop (this should be a fast operation and will reduce the number of pixels to work with)
3. Scale down
4. Equalize the histogram