Processing Android video frame by frame while recording

Question

What I am attempting to do is create an application that adds effects to videos while recording. Is there any way to have a callback method receive the frame, then apply an effect to it and have that recorded. There is currently an application on the Android Market (Videocam Illusion) that claims it is the only application that can do this. Anyone know how Videocam Illusion does this or have some links to possible tutorials outlining video processing for Android?

This is a similar question that is unanswered:

Android preview processing while video recording

Answer 1

Unfortunately, (unless I'm unaware of some other method provided by the API) the way this is done is using a direct stream to the camera and manipulating it by using some sort of Native Code to modify the stream. I've done something similar to this before when I was working on an eyetracker - So I'll tell you how it works basically.

1. Open a stream using the NDK (possibly api, depending on implementations)
2. Modify the bytes of the stream - each frame is sent as a separate packet. You have to grab each packet from the camera, and modify it. You can do a replace of colors, or you can translate. You can also use OpenGL to modify the image entirely by adding things like glass effects.
3. Flatten the images back out
4. send the image over to the view controller to be displayed.

One thing that you have to be mindful of is the load and send of the packets & images happen in about 1/30th of a second for each frame. So the code has to be extremely optimized.