Why transforming normals with the transpose of the inverse of the modelview matrix?

Question

I am working on some shaders, and I need to transform normals.

I read in few tutorials the way you transform normals is you multiply them with the transpose of the inverse of the modelview matrix. But I can't find explanation of why is that so, and what is the logic behind that?

Answer 1

It flows from the definition of a normal.

Suppose you have the normal, N, and a vector, V, a tangent vector at the same position on the object as the normal. Then by definition N·V = 0.

Tangent vectors run in the same direction as the surface of an object. So if your surface is planar then the tangent is the difference between two identifiable points on the object. So if V = Q - R where Q and R are points on the surface then if you transform the object by B:

V' = BQ - BR
   = B(Q - R)
   = BV

The same logic applies for non-planar surfaces by considering limits.

In this case suppose you intend to transform the model by the matrix B. So B will be applied to the geometry. Then to figure out what to do to the normals you need to solve for the matrix, A so that:

(AN)·(BV) = 0

Turning that into a row versus column thing to eliminate the explicit dot product:

[tranpose(AN)](BV) = 0

Pull the transpose outside, eliminate the brackets:

transpose(N)*transpose(A)*B*V = 0

So that's "the transpose of the normal" [product with] "the transpose of the known transformation matrix" [product with] "the transformation we're solving for" [product with] "the vector on the surface of the model" = 0

But we started by stating that transpose(N)*V = 0, since that's the same as saying that N·V = 0. So to satisfy our constraints we need the middle part of the expression — transpose(A)*B — to go away.

Hence we can conclude that:

 transpose(A)*B = identity
 => transpose(A) = identity*inverse(B)
 => transpose(A) = inverse(B)
 => A = transpose(inverse(B))