What you found is probably SetWindowsHookEx which ha nothing to do with the hook we need at all. It can set hooks on the WindowMessage handler of a process.
What we need is called a detours (if we use bytepatches to create a hook)
I guess the detours is easier for beginners and since the windows apis are hotpatchable this will work on any OS version too, no need for the IAT hook for now.
So how does it work?
There is a function prologue for windows apis which looks like this:
74EEEEDA k> 8BFF MOV EDI,EDI
74EEEEDC 55 PUSH EBP
74EEEEDD 8BEC MOV EBP,ESP
As you can see this are exactly 5 bytes thanks to the mov edi, edi
So we will place a jmp instruction there (a jmp is also 5 bytes) and copy the bytes to our trampoline which will first execute those 5 overwritten bytes and ten jumps back past the jump (first 5 bytes) to execute the rest of the bytes.
There are alot of hooking libs out there, but you dont need any special stuff. If you still feel like you can use MS Detours for example (x86 version is free), or PolyHook.
Heres a very simple hooking function which works fine for your needs:
void *CreateDetour(BYTE *src, const BYTE *dst, const int len)
{
BYTE *jmp = (BYTE*)malloc(len + 5);
DWORD dwBack;
VirtualProtect(jmp, len+5, PAGE_EXECUTE_READWRITE, &dwBack);
VirtualProtect(src, len, PAGE_EXECUTE_READWRITE, &dwBack);
Memcpy(jmp, src, len);
jmp += len;
jmp[0] = 0xE9;
*(DWORD*)(jmp + 1) = (DWORD)src + len - (DWORD)jmp - 5;
src[0] = 0xE9;
*(DWORD*)(src + 1) = (DWORD)dst - (DWORD)src - 5;
for(int i = 5; i < len; i++)
src[i] = 0x90;
VirtualProtect(src, len, dwBack, &dwBack);
return(jmp - len);
}
And its usage:
typedef int (WINAPI *tMessageBoxA)(HWND hWnd, LPCTSTR lpText, LPCTSTR lpCaption, UINT uType);
tMessageBoxA oMessageBoxA;
int WINAPI hkMessageBoxA(HWND hWnd, LPCTSTR lpText, LPCTSTR lpCaption, UINT uType)
{
return oMessageBoxA(hWnd, lpText, "Hooked :)", uType);
}
...
oMessageBoxA = (tMessageBoxA)CreateDetour((BYTE*)GetProcAddress(GetModuleHandle("user32.dll"), "MessageBoxA")), (BYTE*)hkMessageBoxA, 5);
MessageBoxA(0, "Test", "Test", 0);
The size is always 5 for windows apis because of the mov edi, edi (hotpatchable image)
