A target program might behave differently if it is being debugged, sometimes this can be very annoying. Also, these behavior deviations can be leveraged by anti-debugging.
IsDebuggerPresent and CheckRemoteDebuggerPresent are well known APIs to tell if a program is attached by a debugger.
0:000> uf KERNELBASE!IsDebuggerPresent KERNELBASE!IsDebuggerPresent: 7512f41b 64a118000000 mov eax,dword ptr fs:[00000018h] 7512f421 8b4030 mov eax,dword ptr [eax+30h] 7512f424 0fb64002 movzx eax,byte ptr [eax+2] 7512f428 c3 ret
CloseHandle would raise an exception under a debugger, as stated by MSDN:
If the application is running under a debugger, the function will throw an exception if it receives either a handle value that is not valid or a pseudo-handle value.
Windows heap manager would use debug heap (note: this has nothing to do with the CRT Debug Heap) if a program was launched from debugger:
OutputDebugString, we've have a dedicated topic on it.
SetUnhandledExceptionFilter, a decent article can be found at debuginfo.com. A simple detouring is to intercept IsDebugPortPresent and return FALSE.
NtSetInformationThread can be used to hide (detach) a thread from debugger.
In addition, the target program can check its own integrity or the integrity of the system.
A few things to mention:
'Digital media application can take advantage of the windows kernel to protect itself from being debugged' - can you please elaborate? What's a digital media application? How can the windows kernel help anti-debugging?
What's a digital media application:
* msdn.microsoft.com/.../ff536256.aspx
* msdn.microsoft.com/.../ff536358.aspx
How can the windows kernel help anti-debugging:
* DRMK.SYS driver has a private contract with the debugging subsystem