How to write opencv mat to gstreamer pipeline?

Question

I want to add some opencv processes to a gstreamer pipeline and then send it over udpsink.

I'm able to read frames from gstreamer like this:

// may add some plugins to the pipeline later
cv::VideoCapture cap("v4l2src ! video/x-raw, framerate=30/1, width=640, height=480, format=RGB ! videoconvert ! appsink");
cv::Mat frame;
while(ture){
  cap >> frame;
  // do some processing to the frame
}

But what can't figure out is how to pass the processed frame to the following pipeline: appsrc ! x264enc ! mpegtsmux ! udpsink host=localhost port=5000

I've tried

cv::VideoWriter writer = cv::VideoWriter("appsrc ! x264enc ! mpegtsmux ! udpsink host=localhost port=5000",  0, (double)30, cv::Size(640, 480), true);
writer << processedFrame;

However, the receiver side receives nothing. (I uses the pipeline $gst-launch-1.0 udpsrc port=5000 ! tsparse ! tsdemux ! h264parse ! avdec_h264 ! videoconvert ! ximagesink sync=false as receiver)

My question is, can I pass processed opencv Mat to a gstreamer pipeline and let it do some encoding, and then send over network through udpsink? If yes, how do I achieve this?

Side question, is there any way I can debug a VideoWriter? Such as checking if frames are actually written into it.

Note that I'm using opencv 2.4.12 and gstreamer 1.2 on ubuntu 14.04.

Any help are great!

EDIT: To provide more info, I tested the following code, and it gave GStreamer Plugin: Embedded video playback halted; module appsrc0 reported: Internal data flow error.

#include <stdio.h>
#include <opencv2/highgui/highgui.hpp>
#include <opencv2/opencv.hpp>

int main(int argc, char *argv[]){
    cv::VideoCapture cap("v4l2src ! video/x-raw, framerate=30/1, width=640, height=480, format=RGB ! videoconvert ! appsink");
    if (!cap.isOpened()) {
        printf("=ERR= can't create capture\n");
        return -1;
    }
    cv::VideoWriter writer;
    // problem here
    writer.open("appsrc ! video/x-raw, framerate=30/1, width=640, height=480, format=RGB ! autovideoconvert ! ximagesink sync=false", 0, (double)30, cv::Size(640, 480), true);
    if (!writer.isOpened()) {
        printf("=ERR= can't create writer\n");
        return -1;
    }

    cv::Mat frame;
    int key;

    while (true) {
        cap >> frame;
        if (frame.empty()) {
            printf("no frame\n");
            break;
        }
        writer << frame;
        key = cv::waitKey( 30 );
    }

    cv::destroyWindow( "video" );
}

Apparently there's something wrong with the appsrc pipeline, but I have no idea what went wrong because the pipeline gst-launch-1.0 v4l2src ! video/x-raw, framerate=30/1, width=640, height=480, format=RGB ! videoconvert ! ximagesink sync=false works fine.

Answer 1

After hours of searching and testing, I finally got the answer. The key is to use only videoconvert after appsrc, no need to set caps. Therefore, a writer pipeline would look like appsrc ! videoconvert ! x264enc ! mpegtsmux ! udpsink host=localhost port=5000.

Following is a sample code that reads images from a gstreamer pipeline, doing some opencv image processing and write it back to the pipeline.

With this method, you can add any opencv process to a gstreamer pipeline easily.

// Compile with: $ g++ opencv_gst.cpp -o opencv_gst `pkg-config --cflags --libs opencv`

#include <stdio.h>
#include <opencv2/highgui/highgui.hpp>
#include <opencv2/opencv.hpp>

int main(int argc, char** argv) {

    // Original gstreamer pipeline: 
    //      == Sender ==
    //      gst-launch-1.0 v4l2src 
    //      ! video/x-raw, framerate=30/1, width=640, height=480, format=RGB 
    //      ! videoconvert
    //      ! x264enc noise-reduction=10000 tune=zerolatency byte-stream=true threads=4
    //      ! mpegtsmux 
    //      ! udpsink host=localhost port=5000
    //      
    //      == Receiver ==
    //      gst-launch-1.0 -ve udpsrc port=5000
    //      ! tsparse ! tsdemux 
    //      ! h264parse ! avdec_h264 
    //      ! videoconvert 
    //      ! ximagesink sync=false

    // first part of sender pipeline
    cv::VideoCapture cap("v4l2src ! video/x-raw, framerate=30/1, width=640, height=480, format=RGB ! videoconvert ! appsink");
    if (!cap.isOpened()) {
        printf("=ERR= can't create video capture\n");
        return -1;
    }

    // second part of sender pipeline
    cv::VideoWriter writer;
    writer.open("appsrc ! videoconvert ! x264enc noise-reduction=10000 tune=zerolatency byte-stream=true threads=4 ! mpegtsmux ! udpsink host=localhost port=9999"
                , 0, (double)30, cv::Size(640, 480), true);
    if (!writer.isOpened()) {
        printf("=ERR= can't create video writer\n");
        return -1;
    }

    cv::Mat frame;
    int key;

    while (true) {

        cap >> frame;
        if (frame.empty())
            break;

        /* Process the frame here */

        writer << frame;
        key = cv::waitKey( 30 );
    }
}

Hope this helps. ;)