How to manipulate texture content on the fly?

Question

I have an iPad app I am working on and one possible feature that we are contemplating is to allow the user to touch an image and deform it.

Basically the image would be like a painting and when the user drags their fingers across the image, the image will deform and the pixels that are touched will be "dragged" along the image. Sorry if this is hard to understand, but the bottom line is that we want to edit the content of the texture on the fly as the user interacts with it.

Is there an effective technique for something like this? I am trying to get a grasp of what would need to be done and how heavy an operation it would be.

Right now the only thing I can think of is to search through the content of the texture based on where was touched and copy the pixel data and do some kind of blend on the existing pixel data as the finger moves. Then reloading the texture with glTexImage2D periodically to get this effect.

Answer 1

There are at least two fundamentally different approaches:

1. Update pixels (i assume this is what you mean in the question)

The most effective technique to change the pixels in the texture is called Render-to-Texture and can be done in OpenGL/OpenGL ES via FBOs. On desktop OpenGL you can use pixel buffer objects (PBOs) to manipulate pixel data directly on GPU (but OpenGL ES does not support this yet).

On unextended OpenGL you can change the pixels in system memory and then update texture with glTexImage2D/glTexSubImage2D - but this is inefficient last resort solution and should be avoided if possible. glTexSubImage2D is usually much faster since it only updates pixel inside the existing texture, while glTexImage2D creates entirely new texture (as a benefit you can change the size and pixel format of the texture). On the other side, glTexSubImage2D allows to update only parts of the texture.

You say that you want it to work with OpenGL ES, so I would propose to do the following steps:

• replace glTexImage2D() with glTexSubImage2D() - if you gain enough performance that's it, just let it be;
• implement render-to-texture with FBOs and shaders - it will require far more work to rewrite your code, but will give even better performance.

For FBOs the code can look like this:

// setup FBO
glGenFramebuffers( 1, &FFrameBuffer );
glBindFramebuffer( GL_FRAMEBUFFER, FFrameBuffer );
glFramebufferTexture2D( GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, YourTextureID, 0 );
glBindFramebuffer( GL_FRAMEBUFFER, 0 );

// render to FBO
glBindFramebuffer( GL_FRAMEBUFFER, FFrameBuffer );
glViewport( 0, 0, YourTextureWidth, YourTextureHeight );
your rendering code goes here - it will draw directly into the texture
glBindFramebuffer( GL_FRAMEBUFFER, 0 );

// cleanup
glBindFramebuffer( GL_FRAMEBUFFER, 0 );
glDeleteFramebuffers( 1, &FFrameBuffer );

Keep in mind that not all pixel formats can be rendered to. RGB/RGBA are usually fine.

2. Update geometry

You can also change the geometry of the object your texture is mapped on. The geometry should be tesselated enough to allow smooth interaction and prevent artifacts to appear. The deformation of geometry can be done via different methods: parametric surfaces, NURBS, patches.