How to build a hook syscall detector Complete Guide by Blackhat Pakistan 2023

Today we will learn about How to build a hook syscall detector.

Windows API hooking

Windows API bowing is a faddy manner used to document and modifies the bearing and discharge flux of syscalls. For illustration, this form is overall used by AV and EDR networks to umpire if a piece of constitution is shady.

additional technically, a hook can be equated to a factor where all or a express band of syscalls parallel as CreateFile(), ReadFile(), OpenProcess(), VirtualAlloc() and so on, could be picked off and oversaw to bear out if the ideal of the deportment is debatable or not.

In item, AV and EDR sellers are grasping better of userland APIs by rape the pictures of the jobs in Windows DLLs, like as kernel32/ kernel base and ntdll( source). The procedure is kind of simple a jmp decree is added to the discharge flood when a special and predefined syscall is invoked, like as CreateFileA().

The workflow is either modified and turned to the agent DLL(EDR.DLL) that will perform documentation jobs and return the torrent to the earliest DLL(KERNEL32.DLL). With this form in place, the agent DLL can bear out the program actions and the input parameters and hence watch all the commission. The redirection process we mentioned earlier is also called a departure or trampoline within the security terrain.

The art of unhooking

The after process of bowing a syscall is unhooking it. The jmp must be refreshed, and we need to know the first state before the hook process. This process is natch
simple, and we just need to check the untouched DLL present on the bit c windows system32kernel32.dll. The almost Virtual Address( RVA) can be obeyed on the line speech Table( EAT), in this case 00022080.

even so, we will get the original address, If we convert the RVA to the uncooked address( the material one) before it’s applied inmemory.However, it’s hooked, If the DLL in memory does n’t match that value.

Also Read:UEFI Boot vs. the MBR/VBR Boot Process-byBlackhat Pakistan 2023

How to build the hook syscall detector

The overall process of detecting a hooked function is composed of several path, Namely:

• Running the exported functions of theNTDLL.DLL.
• Reading the first 4 bytes and checking if it starts with 4c 8b d1 b8. still, the syscall isn’t hooked,
• If it matches. still, the function is presumably hooked with some exceptions mentioned below,
• If it doesn’t match. False cons to consider in this trial

The source law to execute a simple hook syscall sensor can be observed below( credits toired.team blog).

After collecting the source law and executing the binary on a Windows machine with an AV/ EDR installed, we can observe all the hooked functions, with the exception of the NtGetTickCount() call present on the exception list from Figure 3.

Learn about Windows API and hook syscall [How to build a hook syscall detector]

We learned how EDR/ AV systems hook windows API calls in runtime. This process can be fluently reversed by comparing the RVA addresses with the original bones
on the fragment.

With this simple premise in place, erecting a hook syscall sensor is possible, but always keep in mind that some of the detected changes in memory can be false cons from Windows OS. This process can be applied when vicious software is running and detecting changes on the Windows DLLs — a process veritably analogous to that executed by AV/ EDR systems.

Sources:

• Hook detector, Ired.team
• Hook syscalls, GitBook Seguranca-Data