How to optimize ffmpeg w/ x264 for multiple bitrate output files

Question

The goal is to create multiple output files that differ only in bitrate from a single source file. The solutions for this that were documented worked, but had inefficiencies. The solution that I discovered to be most efficient was not documented anywhere that I could see. I am posting it here for review and asking if others know of additional optimizations that can be made.

Source file       MPEG-2 Video (Letterboxed) 1920x1080 @>10Mbps
                  MPEG-1 Audio @ 384Kbps
Destiation files  H264 Video 720x400 @ multiple bitrates
                  AAC Audio @ 128Kbps
Machine           Multi-core Processor

The video quality at each bitrate is important so we are running in 2-Pass mode with the 'medium' preset

VIDEO_OPTIONS_P2 = -vcodec libx264 -preset medium -profile:v main -g 72 -keyint_min 24 -vf scale=720:-1,crop=720:400

The first approach was to encode them all in parallel processes

ffmpeg -y -i $INPUT_FILE $AUDIO_OPTIONS_P2 $VIDEO_OPTIONS_P2 -b:v 250k -threads auto -f mp4 out-250.mp4 &
ffmpeg -y -i $INPUT_FILE $AUDIO_OPTIONS_P2 $VIDEO_OPTIONS_P2 -b:v 500k -threads auto -f mp4 out-500.mp4 &
ffmpeg -y -i $INPUT_FILE $AUDIO_OPTIONS_P2 $VIDEO_OPTIONS_P2 -b:v 700k -threads auto -f mp4 out-700.mp4 &

The obvious inefficiencies are that the source file is read, decoded, scaled, and cropped identically for each process. How can we do this once and then feed the encoders with the result?

The hope was that generating all the encodes in a single ffmpeg command would optimize-out the duplicate steps.

ffmpeg -y -i $INPUT_FILE \
$AUDIO_OPTIONS_P2 $VIDEO_OPTIONS_P2 -b:v 250k -threads auto -f mp4 out-250.mp4 \
$AUDIO_OPTIONS_P2 $VIDEO_OPTIONS_P2 -b:v 500k -threads auto -f mp4 out-500.mp4 \
$AUDIO_OPTIONS_P2 $VIDEO_OPTIONS_P2 -b:v 700k -threads auto -f mp4 out-700.mp4

However, the encoding time was nearly identical to the previous multi-process approach. This leads me to believe that all the steps are again being performed in duplicate.

To force ffmpeg to read, decode, and scale only once, I put those steps in one ffmpeg process and piped the result into another ffmpeg process that performed the encoding. This improved the overall processing time by 15%-20%.

INPUT_STREAM="ffmpeg -i $INPUT_FILE -vf scale=720:-1,crop=720:400 -threads auto -f yuv4mpegpipe -"

$INPUT_STREAM | ffmpeg -y -f yuv4mpegpipe -i - \
$AUDIO_OPTIONS_P2 $VIDEO_OPTIONS_P2 -b:v 250k -threads auto out-250.mp4 \
$AUDIO_OPTIONS_P2 $VIDEO_OPTIONS_P2 -b:v 500k -threads auto out-500.mp4 \
$AUDIO_OPTIONS_P2 $VIDEO_OPTIONS_P2 -b:v 700k -threads auto out-700.mp4

Does anyone see potential problems with doing it this way, or know of a better method?

Answer 1

If you apply the audio/video options to the piped output of the first process, you could save some CPU, since it would exchange 3 encodings to a single one.

ffmpeg -i $INPUT_FILE -vf scale=720:-1,crop=720:400 $AUDIO_OPTIONS_P2 $VIDEO_OPTIONS_P2  -f yuv4mpegpipe -\
    | ffmpeg -y -f yuv4mpegpipe -i - \
          -b:v 250k out-250.mp4 \
          -b:v 500k out-500.mp4 \
          -b:v 700k out-700.mp4

This is the recommended way for older versions of ffmpeg. There's a newer method (didn't test it) available since earlier this month: http://ffmpeg.org/trac/ffmpeg/wiki/Creating%20multiple%20outputs