Signed angle between two 3D vectors with same origin within the same plane

Question

What I need is a signed angle of rotation between two vectors Va and Vb lying within the same 3D plane and having the same origin knowing that:

1. The plane contatining both vectors is an arbitrary and is not parallel to XY or any other of cardinal planes
2. Vn - is a plane normal
3. Both vectors along with the normal have the same origin O = { 0, 0, 0 }
4. Va - is a reference for measuring the left handed rotation at Vn

The angle should be measured in such a way so if the plane would be XY plane the Va would stand for X axis unit vector of it.

I guess I should perform a kind of coordinate space transformation by using the Va as the X-axis and the cross product of Vb and Vn as the Y-axis and then just using some 2d method like with atan2() or something. Any ideas? Formulas?

Answer 1

Use cross product of the two vectors to get the normal of the plane formed by the two vectors. Then check the dotproduct between that and the original plane normal to see if they are facing the same direction.

angle = acos(dotProduct(Va.normalize(), Vb.normalize()));
cross = crossProduct(Va, Vb);
if (dotProduct(Vn, cross) < 0) { // Or > 0
  angle = -angle;
}