What is the recommended way to hook Win32 APIs for a commmercial application?
What is your recommendation for an API hooking library or code to be used in a commercial application?
I have looked at Microsoft Detours which seems to be very good, but definitely is out of budget for the profit I am expecting out of my application.
Is there any library that offers compatibility across WinXP and Vista (and Windows 7 if not too much to ask!)? Is there anyone with past experience in using such a library in a commercial product?
Answer
API hooking in Win32 isn't really possible in a system-wide sense. You can approximate it by injecting a DLL into each process and then patching each process from within. You can either use IAT patching (where you patch the calling binary) or a Detours-style patch (where you patch the callee).
Patching the caller (IAT patching) means that you need to enumerate every DLL that is loaded in the process and patch each one separately. You also would need to hook LoadLibrary in order to patch any new DLLs that are loaded on-the-fly.
Patching the callee (Detours) has the advantage that you only need to patch one location to have the hook apply to the entire process.
You have to do the per-process patching even if you're hooking APIs from shared system DLLs; the OS will invoke copy-on-write whereby when you patch the system DLL, the process is given a private copy to be patched.
DLL injecting gets to be a bit nasty, and again there are several techniques: AppInit_DLLs, which only works for processes that load USER32.DLL (and has several new restrictions in Vista and Windows 7), using SetWindowsHookEx, or by using CreateRemoteThread. Integrity levels in Vista and Windows 7 make it more difficult to inject into processes system-wide. Your app will need to run with administrator privileges and a high integrity level to be able to successfully pull it off.
Another technique is to hook the system services in kernel-mode. This requires writing a device driver, but it is basically the technique that Sysinternals Process Monitor uses (or at least did, once). This is a problem on 64-bit Vista and Win7 because of PatchGuard and the driver signing requirements. You can monitor some file system activity by using file system filter drivers.