Debugging Direct3D Fullscreen App

[ArC]

I'm trying to "debug" a Direct3D game that runs in fullscreen mode with Olly.
The problem is that once a breakpoint is triggered there's no way to get to the debugger window in order to start tracing or continue execution.
I know this is a common problem when debugging Direct3D fullscreen apps.
I'd like to know whether there're ways to debug such a fullscreen app. I read sth about a second monitor or remote debugging but they're out of question since I don't have a second monitor and OllyDbg doesn't support Remote Debugging. One way I found was to run the app in windowed mode. The game itself doesn't support windowed mode so are there tools available I can use to force the game to run in windowed mode? I know that 3D Analyzer can do this but the game quits immediately when I start it with the "force windowed mode" option enabled.

Thx

[ricnar456]

i´m debugging a direct x now with olly.

i use olly and have good results the method of work is very limited but i have good results.

1)never put a breakpoint of hardware bpx, this freeze your machine.
2)for use breakpints i made a litlle script in this form, i run the game, alt mas tab for switch to olly, run the script


var aux
start:

eob break
run

break:
log eax
log ebx
log ecx
log edx
log esp
log ebp
log esi
log edi
log eip
mov aux,esp
log [aux]
add aux,4
log [aux]
add aux,4
log [aux]
add aux,4
log [aux]
add aux,4
log [aux]
add aux,4
log [aux]
add aux,4
log [aux]
mov aux,0

jmp start

this make olly dont stop in the HE and continue running the game without freeze, next you can go out of the game woth alt mas tab and look the values stored in the log, plus if is a repetitive address the screen will swith to olly without freeze and you can pause the script and pause olly and continue working.
Is a very hard method but work for me

if you can make a good script than change eip and execute a loop in the program, your screen don´t freeze and you can switch to olly with alt + tab and return to the adress of the HE and continue tracing:

I make all dirty tricks how this and with more work, but always can crack direct X games with olly, only you need prevent olly stop completely in a BP o HE while you have the screen of the game in the monitor.
if you work without stop olly, and you can switch to olly, you can continue tracing.

Ricardo Narvaja

[ArC]

Thx interesting tips. I'll try them.

well directdraw means it aslo uses vtable and alomost all calls would be of
the kind call[reg32+const] in these situations using log breakpoints (never pause) log always or log on condition log expression always or log expression on condition log arguments always or on condition gets you almost all the info with alt+tab and application runnnig

also if you set a breakpoint and it froze you can use taskmanager (open it earlier and alttab to it or use ctrl+shift+esc to bring it up and then
use bring to front olly with task manager you can even trace live
ok the graphics will be garish and some other problems
bit you can single step

Try to use SoftIce (May be in FullScreen Mode).

Best and simpliest - use another graphic card and monitor - its not so expenisive today.

[niom]

some time ago i wrote a dll, that simply hooks IDirect3DDevice9::CreateDevice

and when this function is called by the game, i set windowed bits in the D3DPRESENT_PARAMETERS param

this works for many games

Then we are back to the same problem of finding ID3DDevice9, which of course is not terribly difficult. Once the ID3DDevice9 pointer is found, the rest is simply cake.

[niom]

HRESULT __stdcall myCreateDevice(CREATEDEVICE tramp, IDirect3D9 *d3d, UINT Adapter, D3DDEVTYPE DeviceType, HWND hFocusWindow, DWORD BehaviorFlags, D3DPRESENT_PARAMETERS *pPresentationParameters, IDirect3DDevice9 **ppReturnedDeviceInterface)
{
	HWND wnd;

	wnd = pPresentationParameters->hDeviceWindow;

	SetWindowPos(wnd, 0, 10, 10, 640, 480, SWP_NOZORDER);
	SetWindowLong(wnd, GWL_STYLE, WS_VISIBLE);
	SetWindowLong(wnd, GWL_EXSTYLE, 0);

	memset(pPresentationParameters, 0, sizeof(D3DPRESENT_PARAMETERS));
	pPresentationParameters->BackBufferCount = 2;
	pPresentationParameters->Windowed = TRUE;
	pPresentationParameters->SwapEffect =  D3DSWAPEFFECT_FLIP;
	pPresentationParameters->BackBufferFormat = D3DFMT_UNKNOWN;
	pPresentationParameters->hDeviceWindow = wnd;
	pPresentationParameters->EnableAutoDepthStencil = TRUE;
	pPresentationParameters->AutoDepthStencilFormat = D3DFMT_D16;
	pPresentationParameters->PresentationInterval = D3DPRESENT_INTERVAL_IMMEDIATE;

	return tramp(d3d, Adapter, DeviceType, hFocusWindow, BehaviorFlags, pPresentationParameters, ppReturnedDeviceInterface);
}

IDirect3D9 *WINAPI Direct3DCreate9(UINT SDKVersion)
{
	IDirect3D9 *result;
	DWORD *vtable;
	HMODULE dll;
	DIRECT3DCREATE9 orgDirect3DCreate9;

	if ((dll = LoadLibrary("\\windows\\system32\\d3d9.dll")) == NULL) return NULL;

	if ((orgDirect3DCreate9 = (DIRECT3DCREATE9)GetProcAddress(dll, "Direct3DCreate9")) == NULL) return NULL;

	if ((result = orgDirect3DCreate9(SDKVersion)) == NULL) return NULL;

	vtable = *(DWORD **)result;

	hookPre(myCreateDevice, (void *)vtable[16]);		//vtable[16]: d3d->CreateDevice()

	return result;
}