In reversing, anti-debugging tricks have always been a highly interesting matter. Since the migration towards x64 hardware and OS'es, some things have changed though.
The other day, I came across a x64 software which was always fake detecting debugging on a certain test system. Diving into the matter and circumventing all anti-debugging tricks under debugger, it worked fine. The reason of faillure outside debugger proved to be the well-known rep stos/movs trick.
Code:
Example code
t1 equ goodboy-badboy-2
new: db 0EBh,01,81h,0EBh,t1
; cut //
lea rsi,[new]
lea rdi,[here]
mov rdx,[rdi]
mov rcx,3
here:
rep movsw
badboy:
mov r9,30h
lea r8,[DebugStatus]
lea rdx,[DbgFoundText]
xor rcx,rcx
invoke MessageBox ; badboy!
jmp Exit
goodboy:
mov r9,40h
lea r8,[DebugStatus]
lea rdx,[DbgNotFoundText]
xor rcx,rcx
invoke MessageBox ; goodboy!
; cut //
The rep stos/movs trick does not need further explaining since everybody knows this one since 16 bit. However, be warned not to use it anymore on x64. For testing, I attached an exe. Single step it F7 (F8 on the messagebox call) and it will always detect you, however I'm sure that a small percentage -having the newest x64 CPU technology- will get fake detected outside debugger!
Carpe Diem,
lena151.