How can I perform hardware-accelerated H.264 encoding and decoding for streaming?

Question

I am able to get the RGBA frame data from the camera, and I want to encode it in H.264 format. I've used FFmpeg to encode and decode H.264 video, but at a frame size of 640x480 it's too slow for my needs.

I'd like to use hardware acceleration to speed up the encoding and decoding, so how would I do that?

Also, I need to be able to stream the encoded video across the network and decode it on the other end. How can this be done?

Answer 1

If you want to do hardware-accelerated video encoding and decoding of H.264 video on iOS, the only way to go is AVFoundation. Don't use third-party libraries for the encoding or decoding, because they all currently are CPU-bound, and are much slower than what you get from AVFoundation. The one reason to use a third-party encoder or decoder would be if you are working with a format not supported in iOS by default.

For hardware-accelerated decoding, you'll want to use an AVAssetReader instance (or one of the player classes for pure playback). With an AVAssetReader, I regularly get 2X or higher playback speeds for reading H.264-encoded video, so the iOS devices use some pretty good hardware acceleration for that.

Similarly, for accelerated encoding, you'll use an AVAssetWriter. There are some tricks to getting AVAssetWriter to encode at the best speed (feeding in BGRA frames, using a pixel buffer pool, using the iOS 5.0 texture caches if reading from OpenGL ES), which I describe in detail within this answer.

If you want to see some code that uses the fastest paths I've found for accelerated encoding and decoding, you can look at my open source GPUImage framework, which, unlike the one linked by Anastasia, is totally free to use.