How to set decode pixel format in libavcodec?

Question

I decode video via libavcodec, using the following code:

//Open input file
if(avformat_open_input(&ctx, filename, NULL, NULL)!=0)
    return FALSE; // Couldn't open file
if(avformat_find_stream_info(ctx, NULL)<0)
    return FALSE; // Couldn't find stream information
videoStream = -1;
//find video stream
for(i=0; i<ctx->nb_streams; i++)
{       
    if((ctx->streams[i])->codec->codec_type==AVMEDIA_TYPE_VIDEO)
    {
        videoStream=i;
        break;
    }
}
if (videoStream == -1)
    return FALSE; // Didn't find a video stream
video_codec_ctx=ctx->streams[videoStream]->codec;
//find decoder
video_codec=avcodec_find_decoder(video_codec_ctx->codec_id);
if(video_codec==NULL)
    return FALSE; // Codec not found
if(avcodec_open(video_codec_ctx, video_codec)<0)
    return -1; // Could not open codec
video_frame=avcodec_alloc_frame();
scaled_frame=avcodec_alloc_frame();
static struct SwsContext *img_convert_ctx; 
if(img_convert_ctx == NULL) 
{
      int w = video_codec_ctx->width;
      int h = video_codec_ctx->height;
      img_convert_ctx = sws_getContext(w, h, 
                        video_codec_ctx->pix_fmt, 
                        w, h, dst_pix_fmt, SWS_BICUBIC, 
                        NULL, NULL, NULL);
      if(img_convert_ctx == NULL) {
        fprintf(stderr, "Cannot initialize the conversion context!\n");
        return FALSE;
      }
}
while(b_play) 
{
    if (av_read_frame(ctx, &packet) < 0)
    {
        break;
    }
    if(packet.stream_index==videoStream) {
    // Decode video frame   
        avcodec_decode_video2(video_codec_ctx, video_frame, &frameFinished,
                         &packet);
        // Did we get a video frame?
        if(frameFinished) 
        {
            if (video_codec_ctx->pix_fmt != dst_pix_fmt)
            {                       
                if (video_codec_ctx->pix_fmt != dst_pix_fmt)            
                     sws_scale(img_convert_ctx, video_frame->data, 
                              video_frame->linesize, 0, 
                              video_codec_ctx->height, 
                              scaled_frame->data, scaled_frame->linesize);              
            }           
        }
}
av_free_packet(&packet);
}

The code works correctly, but it is necessary to convert each frame to the required format. Is it possible to set the pixel format for decoding to get the correct format without sws_scale?

Many thanks for your answers.

Answer 1

ffmpeg's AVCodec instances (static decoder "factory" objects) each define an array of pixel formats that they support, terminated by the value -1.

The AVCodecContext (decoder instance) objects have a callback function pointer called get_format: it is a function pointer in that structure.

This callback function is called, at some point in the codec initialization, with the AVCodec factory object's array of supported formats, and the callback is supposed to choose one of the formats from that array (kind of like "pick a card, any card") and return that value. The default implementation of this get_format callback is a function called avcodec_default_get_format. (This is installed avcodec_get_context_defaults2). This default function implements the "pick a format" logic quite simply: it chooses the first element of the array which isn't a hardware-accel-only pixel format.

If you want the codec to work with a different pixel format, what you can do is install your own get_format callback into the context object. However, the callback must return one of the values in the array (like choosing from a menu). It cannot return an arbitrary value. The codec will only support the formats that it specifies in the array.

Walk the array of available formats and pick the best one. If you're lucky, it's the exact one you actually want and the sws_scale function won't have to do pixel format conversion. (If, additionally, you don't request to scale or crop the picture, sws_scale should recognize that the conversion is a noop.)