OpenGL - mask with multiple textures

Question

I have implemented masking in OpenGL according to the following concept:

• The mask is composed of black and white colors.
• A foreground texture should only be visible in the white parts of the mask.
• A background texture should only be visible in the black parts of the mask.

I can make the white part or the black part work as supposed by using glBlendFunc(), but not the two at the same time, because the foreground layer not only blends onto the mask, but also onto the background layer.

Is there anyone who knows how to accomplish this in the best way? I have been searching the net and read something about fragment shaders. Is this the way to go?

Answer 1

This should work:

glEnable(GL_BLEND);
// Use a simple blendfunc for drawing the background
glBlendFunc(GL_ONE, GL_ZERO);
// Draw entire background without masking
drawQuad(backgroundTexture);
// Next, we want a blendfunc that doesn't change the color of any pixels,
// but rather replaces the framebuffer alpha values with values based
// on the whiteness of the mask. In other words, if a pixel is white in the mask,
// then the corresponding framebuffer pixel's alpha will be set to 1.
glBlendFuncSeparate(GL_ZERO, GL_ONE, GL_SRC_COLOR, GL_ZERO);
// Now "draw" the mask (again, this doesn't produce a visible result, it just
// changes the alpha values in the framebuffer)
drawQuad(maskTexture);
// Finally, we want a blendfunc that makes the foreground visible only in
// areas with high alpha.
glBlendFunc(GL_DST_ALPHA, GL_ONE_MINUS_DST_ALPHA);
drawQuad(foregroundTexture);

This is fairly tricky, so tell me if anything is unclear.

Don't forget to request an alpha buffer when creating the GL context. Otherwise it's possible to get a context without an alpha buffer.

Edit: Here, I made an illustration.

Edit: Since writing this answer, I've learned that there are better ways to do this:

• If you're limited to OpenGL's fixed-function pipeline, use texture environments
• If you can use shaders, use a fragment shader.

The way described in this answer works and is not particularly worse in performance than these 2 better options, but is less elegant and less flexible.