Get frame from video with libvlc smem and convert it to opencv Mat. (c++)

Question

[UPDATED WITH PARTIAL ANSWER]
Here is my code:

void cbVideoPrerender(void *p_video_data, uint8_t **pp_pixel_buffer, int size) {
    // Locking
    imageMutex.lock();
    videoBuffer = (uint8_t *)malloc(size);
    *pp_pixel_buffer = videoBuffer;
}  
void cbVideoPostrender(void *p_video_data, uint8_t *p_pixel_buffer
      , int width, int height, int pixel_pitch, int size, int64_t pts) {
   // Unlocking
   imageMutex.unlock();
   Mat img = Mat(Size(width,height), CV_8UC3, p_pixel_buffer);
   //cvtColor(img,img,CV_RGB2BGR);
}
int main(int argc, char ** argv)
{
   libvlc_instance_t * inst;
   char smem_options[1000];
   sprintf(smem_options
      , "#transcode{vcodec=RV24}:smem{"
         "video-prerender-callback=%lld,"
         "video-postrender-callback=%lld,"
         "video-data=%lld,"
         "no-time-sync},"
      , (long long int)(intptr_t)(void*)&cbVideoPrerender
      , (long long int)(intptr_t)(void*)&cbVideoPostrender //This would normally be useful data, 100 is just test data
      , (long long int)200 //Test data
      );
    const char * const vlc_args[] = {
              "-I", "dummy", // Don't use any interface
              "--ignore-config", // Don't use VLC's config
              "--extraintf=logger", // Log anything
              "--verbose=1", // Be verbose
              "--sout", smem_options // Stream to memory
               };

    // We launch VLC
    inst = libvlc_new(sizeof(vlc_args) / sizeof(vlc_args[0]), vlc_args);
...
return 0;
}

QUESTION UPDATED
I checked my two callback functions seem correctly executed.
_What kind of data does RV32 exactly output ? Does it fit the CV_8U3C (unsigned 8bits int 3 channel required here? _Do I need to add a step to my Mat class ? (step – Number of bytes each matrix row occupies)
UPDATED2
I changed RV32 to RV24 which makes more sense. I add cvtColor cause the Mat matrix seems to need BGR pixel and not RGB but still the image is not display correctly.
_Is there a vcodec which would give me a YUV format as output so I can test the pixel data before trying to output an opencv::Mat img ?
[EDIT OUTPUT IMG] (By changing vlc type to CV_8UC4 four channel (dont know why) we can almost see the frame but in really poor quality why is that ?
[SOLUTION]
I found out that the images at the beginning of my video were of poor quality that's why my Mat imshow() showed me such ugly thing the code above should work now (Apperently no need for cvtColor)

Answer 1

First, a quick warning: starting with VLC2.2 (current git version, to be released soon), the size parameter is a size_t. There's no API for smem (yet? hopefully this will change), which sucks, so this would silently break your application.

Then, a quick comment about the "data" parameter: it's supposed to hold what you need to do your processing. That being a pointer to a struct, an instance of a class, you name it. I strongly doubt passing a long long would work on a 32bits machine, since you'd be forcing 64 bits in something which can only contain 32. What you should do is to declare a struct, and store what you need into it. Here, a good example could be:

struct MyParamStruct
{
    YourMutexType imageMutex; // Here mutex is not a global variable anymore
    int otherParam; // You can use this to store the value 200 that you were passing before
};
//...

// Init the struct somewhere
MyParamStruct* param = new MyStructParam;
param->otherParam = 200;
//...

sprintf(smem_options
      , "#transcode{vcodec=h264}:smem{"
         "video-prerender-callback=%lld,"
         "video-postrender-callback=%lld,"
         "video-data=%lld,"
         "no-time-sync},"
      , (long long int)(intptr_t)(void*)&cbVideoPrerender
      , (long long int)(intptr_t)(void*)&cbVideoPostrender //This would normally be useful data, 100 is just test data
      , (long long int)(intptr_t)(void*)param
      );

About the mutex usage, it looks good to me. Actually it seems that you don't have any concurrency issue here, as you synchronously allocate a new buffer for each frame. If you were using a preallocated buffer each time, you would need to consider locking when exiting the postrender function.

In fact I'm not even sure about what is exactly the void pointer p_video_data.

That depends on your image format. For H264, it would depend on the pixel format that will be output by the decoder. Since you're asking for H264 output, it's quite likely you'll get a planar pixel format, though the exact type would depend on your H264 profile.

If you're expecting rawdata as a result (which seems to be the case, as CV_8UC3 seems to be refering to a 3 channels raw image, after a quick glance at google), I'd recommend you switch to RV32: #transcode{vcodec=RV32}

What you need to pass to the transcode module is your output fourcc, VLC will deal with the input for you :)

Update

I have no idea if the Mat class takes the ownership of your pointer, but you might want to check that as well.

Update 2

To answer your further question about what is RV32:

/* 24 bits RGB */
#define VLC_CODEC_RGB24           VLC_FOURCC('R','V','2','4')
/* 24 bits RGB padded to 32 bits */
#define VLC_CODEC_RGB32           VLC_FOURCC('R','V','3','2')
/* 32 bits RGBA */
#define VLC_CODEC_RGBA            VLC_FOURCC('R','G','B','A')

If you expect 3 bytes only, then you probably should RV24 a try! I should probably have suggested that from the beginning, since the 8CU3 definitely suggests 3 bytes only...