Hooking DirectX EndScene from an injected DLL
- by Etan
I want to detour EndScene from an arbitrary DirectX 9 application to create a small overlay. As an example, you could take the frame counter overlay of FRAPS, which is shown in games when activated.
I know the following methods to do this:
Creating a new d3d9.dll, which is then copied to the games path. Since the current folder is searched first, before going to system32 etc., my modified DLL gets loaded, executing my additional code.
Downside: You have to put it there before you start the game.
Same as the first method, but replacing the DLL in system32 directly.
Downside: You cannot add game specific code. You cannot exclude applications where you don't want your DLL to be loaded.
Getting the EndScene offset directly from the DLL using tools like IDA Pro 4.9 Free. Since the DLL gets loaded as is, you can just add this offset to the DLL starting address, when it is mapped to the game, to get the actual offset, and then hook it.
Downside: The offset is not the same on every system.
Hooking Direct3DCreate9 to get the D3D9, then hooking D3D9-CreateDevice to get the device pointer, and then hooking Device-EndScene through the virtual table.
Downside: The DLL cannot be injected, when the process is already running. You have to start the process with the CREATE_SUSPENDED flag to hook the initial Direct3DCreate9.
Creating a new Device in a new window, as soon as the DLL gets injected. Then, getting the EndScene offset from this device and hooking it, resulting in a hook for the device which is used by the game.
Downside: as of some information I have read, creating a second device may interfere with the existing device, and it may bug with windowed vs. fullscreen mode etc.
Same as the third method. However, you'll do a pattern scan to get EndScene.
Downside: doesn't look that reliable.
How can I hook EndScene from an injected DLL, which may be loaded when the game is already running, without having to deal with different d3d9.dll's on other systems, and with a method which is reliable? How does FRAPS for example perform it's DirectX hooks?
The DLL should not apply to all games, just to specific processes where I inject it via CreateRemoteThread.
