I'd like to get an idea what kind of math is typically necessary for 3D game engine programming? Any specific math (such as vector geometry) or calculation algorithms (such as fast fourier transforms), or is this all abstracted away with DirectX/OpenGL so that highly complex math isn't really necessary any more?
Linear Algebra! Lots of lots of Linear Algebra!
Here are just classes and example situations where you need them
You should start with Vector and Matrix as they will be used everywhere in the engine (graphics, physics, AI, etc.)