Hooking DirectX EndScene from an injected DLL

Question

I want to detour EndScene from an arbitrary DirectX 9 application to create a small overlay. As an example, you could take the frame counter overlay of FRAPS, which is shown in games when activated.

I know the following methods to do this:

1. Creating a new d3d9.dll, which is then copied to the games path. Since the current folder is searched first, before going to system32 etc., my modified DLL gets loaded, executing my additional code.
   Downside: You have to put it there before you start the game.

2. Same as the first method, but replacing the DLL in system32 directly.
   Downside: You cannot add game specific code. You cannot exclude applications where you don't want your DLL to be loaded.

3. Getting the EndScene offset directly from the DLL using tools like IDA Pro 4.9 Free. Since the DLL gets loaded as is, you can just add this offset to the DLL starting address, when it is mapped to the game, to get the actual offset, and then hook it.
   Downside: The offset is not the same on every system.

4. Hooking Direct3DCreate9 to get the D3D9, then hooking D3D9->CreateDevice to get the device pointer, and then hooking Device->EndScene through the virtual table.
   Downside: The DLL cannot be injected, when the process is already running. You have to start the process with the CREATE_SUSPENDED flag to hook the initial Direct3DCreate9.

5. Creating a new Device in a new window, as soon as the DLL gets injected. Then, getting the EndScene offset from this device and hooking it, resulting in a hook for the device which is used by the game.
   Downside: as of some information I have read, creating a second device may interfere with the existing device, and it may bug with windowed vs. fullscreen mode etc.

6. Same as the third method. However, you'll do a pattern scan to get EndScene.
   Downside: doesn't look that reliable.

How can I hook EndScene from an injected DLL, which may be loaded when the game is already running, without having to deal with different d3d9.dll's on other systems, and with a method which is reliable? How does FRAPS for example perform it's DirectX hooks? The DLL should not apply to all games, just to specific processes where I inject it via CreateRemoteThread.

Answer 1

You install a system wide hook. (SetWindowsHookEx) With this done, you get to be loaded into every process.

Now when the hook is called, you look for a loaded d3d9.dll.

If one is loaded, you create a temporary D3D9 object, and walk the vtable to get the address of the EndScene method.

Then you can patch the EndScene call, with your own method. (Replace the first instruction in EndScene by a call to your method.

When you are done, you have to patch the call back, to call the original EndScene method. And then reinstall your patch.

This is the way FRAPS does it. (Link)

---

You can find a function address from the vtable of an interface.

So you can do the following (Pseudo-Code):

IDirect3DDevice9* pTempDev = ...;
const int EndSceneIndex = 26 (?);

typedef HRESULT (IDirect3DDevice9::* EndSceneFunc)( void );

BYTE* pVtable = reinterpret_cast<void*>( pTempDev );
EndSceneFunc = pVtable + sizeof(void*) * EndSceneIndex;

EndSceneFunc does now contain a pointer to the function itself. We can now either patch all call-sites or we can patch the function itself.

Beware that this all depends on the knowledge of the implementation of COM-Interfaces in Windows. But this works on all windows versions (either 32 or 64, not both at the same time).