IDA SDK C++ tricks and tips

[chants]

IDA offers a good amount of info about functions but its SDK is basically undocumented for accessing some of it. Fortunately ida64.dll can be decompiled and the little info the SDK docs understood.

Functions: func_t offers function details including members:

      uval_t frame;        ///< netnode id of frame structure - see frame.hpp
...
        // the following fields should not be accessed directly:

      uint32 pntqty;       ///< number of SP change points
      stkpnt_t *points;    ///< array of SP change points.
                           ///< use ...stkpnt...() functions to access this array.

      int regvarqty;       ///< number of register variables (-1-not read in yet)
                           ///< use find_regvar() to read register variables
      regvar_t *regvars;   ///< array of register variables.
                           ///< this array is sorted by: start_ea.
                           ///< use ...regvar...() functions to access this array.

      int llabelqty;       ///< number of local labels
      llabel_t *llabels;   ///< local labels.
                           ///< this array is sorted by ea.
                           ///< use ...llabel...() functions to access this array.

      int regargqty;       ///< number of register arguments
      regarg_t *regargs;   ///< unsorted array of register arguments.
                           ///< use ...regarg...() functions to access this array.

      int tailqty;         ///< number of function tails
      range_t *tails;      ///< array of tails, sorted by ea.
                           ///< use func_tail_iterator_t to access function tails.

For regargs for example there is no way to read except through this do not access structure member. Most of them can only be queried on a per address or per item basis and not obtaining all for the function simultaneously as the structure hints should be easily possible. Further these are stored packed in netnodes and are not populated except on demand. Although the quantity fields will have correct values, the actual data will be null pointers waiting for a load on first use. So how to invoke this first use?

	size_t num = get_func_qty();
	for (size_t i = 0; i < num; i++) {
		func_t* f = getn_func(i);
		get_frame(f->start_ea); //load frame into internal pointer table
		find_regvar(f, f->start_ea, nullptr); //load regvars
		get_spd(f, f->start_ea); //load points
		read_regargs(f); //load regargs
		get_llabel(f, ea); //load llabels
		func_tail_iterator_t fti(pfn); //load tails
	}

get_frame(f/ea) is the only and necessary way to immediately load the frame information from the database. It is not stored in the function structure but the pointer can be looked up without being loaded/unpacked from db again. If the functions frame member is BADNODE then obviously this will return a null pointer and there is no frame for the function.

find_regvar(f, ea, _) although it is undocumented can accept a nullptr for the third argument and regardless it will always load the regvars member on first use. Presumably one could find the register variables by giving every possible canonical register name via enumeration of ph.reg_names[] array but this would be over doing it as regvars can be populated with a nullptr and accessing the array directly.

get_spd(f, ea), get_effective_sp(f, ea), get_sp_delta(f, ea) will always populate the points array if you pass the function pointer on first use. Although IDA claims a null pointer is possible, this function will always return 0 until it is called at least once on the function in question with that filled in. It does not call get_func(ea) for you like you would think. Totally misdocumented feature. recalc_spd(ea) will not always work either if one of the get_*sp* functions has not been used first - it works in some cases depending on which address in the function is used. If the ea for recalc_spd is part of a function chunk with no sp changes - it will not do anything leaving the points null. It actually walks through instructions in the particular function chunk/tail entry of the ea only so if there is a stack change in that exact one it will populate but its highly unlikely one would know this in advance. With that in mind, the stack points could be enumerated by going through every assembly instruction in the function using the appropriate enumeration functions, and querying get_*sp* functions to gather the info. But since we know the function has the array already such intensive searching is fortunately not necessary.

read_regargs(f) populates the regargs and is the only way to do so. They have to be accessed directly.

Now you have a regarg_t structure so what about this strange type_t *type; member? Mostly we want tinfo_t not type_t raw data.

Well this again is a poorly documented and tricky to access structure. Obviously we do not want to parse through it ourselves as its not SDK version portable, and its packed in a really detailed format. But if you try to load it directly it will cause corruption and a crash or bad data.

if (f->regargs[i].type != nullptr) {
	tinfo_t ti; //The type information is internally kept as an array of bytes terminated by 0.
	qtype typeForDeser;
	typeForDeser.append(f->regargs[i].type);
	ti.deserialize(get_idati(), &typeForDeser); //will free buffer passed!
}

The underlying deserialize API function looks like and the pointer to a pointer for the type_t member is notable - you must pass an IDA API compatible buffer:

decl bool ida_export deserialize_tinfo(tinfo_t *tif, const til_t *til, const type_t **ptype, const p_list **pfields, const p_list **pfldcmts);

As for function tails, these are relatively well documented right in funcs.hpp, finally having an iterator and not needing to use the internal access (although you can't index them but must enumerate them, a very small downside, in this case the SDK method is recommended):

      func_tail_iterator_t fti(pfn);
      for ( bool ok=fti.first(); ok; ok=fti.next() )
        const range_t &a = fti.chunk();

However if you want direct access to the array, and want preloading of it, this will do the trick (or anything that ultimately invokes func_tail_iterator_set(func_tail_iterator_t*, f, ea)):

      func_tail_iterator_t fti(pfn);

Finally for labels, a different situation occurs. The actual llabel_t structure for a local label and the 3 associated functions set_llabel, get_llabel_ea and get_llabel are all mentioned with caveat: "These are LOW LEVEL FUNCTIONS. When possible, they should not be used. Use high level functions from <name.hpp>". Instead set_name, get_name_ea and get_name respectively should be used. And yes named could be enumerated by doing a walk through each assembly instruction or using various next_* searching functions. But whatever populating for direct access to all function labels quick and easy? This one seems to not be in ida64.dll but is actually provided as a .lib method. But the lib will just forward these to the recommended functions so it does not really matter.

So get_llabel(f, ea) or get_name(f, ea) will reliably do the job and there is no other suitable candidate though get_llabel_ea(f, nullptr) and get_name_ea(f, nullptr) might work if they accept a null pointer for the string.

IDA provides a very practical SDK for a very practical purpose. With a very big support price tag for it, it is no wonder there is poor documentation and lack of consistency. Working on the clock, it would be easier to justify a paper trail of support emails asking about these type of details rather than painstakingly tracking them out through trial and error or reverse engineering SDK lib files or the ida64.dll.

Perhaps these are part of IDA Pro's trade secrets, if you were to think of it as such.

It was quite difficult to get these simple things working correctly as none of them are properly documented. I hope this helps someone on their own IDA SDK development efforts.

In a multithreaded plugin, preloading everything is actually a necessity to minimize thread safety issues around various unsafe database access to race conditions. By preloading all of these structures, even if you want to access them using nicer API methods, it will not cause any database calls. Unfortunately the IDA SDK is designed for single threaded use mostly so doing a dump or lots of UI thread callbacks would be the only other options. But this idea will mitigate all contention except those caused by the user's own interaction changing data structures in use.

[chants]

Another interesting function is execute_sync with MFF_NOWAIT which is needed when a thread doing work may need to be interrupted to prevent deadlock.

THREAD_SAFE inline int execute_sync(exec_request_t &req, int reqf) { return callui(ui_execute_sync, &req, reqf).i; }

We see a exec_request_t object reference is passed, and then a pointer to this is passed along. The return value is the ID for cancelling:

THREAD_SAFE inline bool cancel_exec_request(int req_id)

Now if you want to cancel the request, such as in an interruption case, likely crashes will occur. This is because for whatever inexplicable reason, cancel_exec_request is calling delete on the exec_request_t pointer. So if you did not allocate it with new but just as a stack variable, it will be trying to free stack memory causing an obvious crash.

To deal with this, the best I could come up with was using both a semaphore to track the exec request completion, along with a termination flag and a mutex around that to prevent race conditions:
Global (hopefully stored somewhere referenceable from a singular global object):

		bool exiting = false;
		int uiExecutingTask = -1;
		qmutex_t qm = qmutex_create();

Main thread termination:

		{ //must be a locked unit otherwise race condition can occur
			qmutex_locker_t lock(qm);
			exiting = true;
			if (uiExecutingTask != -1) {
				cancel_exec_request(uiExecutingTask);
			}
		}

	struct execFunctor : public exec_request_t
	{
		std::function<void()> fun;
		qsemaphore_t finishSem;
		execFunctor(std::function<void()> f, qsemaphore_t qs) : fun(f), finishSem(qs) {}
		virtual ~execFunctor(void) override {}
		/// Callback to be executed.
		/// If this function raises an exception, execute_sync() never returns.
		virtual int idaapi execute(void)
		{
			fun();
			qsem_post(finishSem);
			return 0;
		}
	};
	void executeOnMainThread(std::function<void()> fun)
	{
		qsemaphore_t finishSem = qsem_create(nullptr, 0);
		execFunctor* ef = new execFunctor(fun, finishSem);
		{ //must be a locked unit otherwise race condition can occur
			qmutex_locker_t lock(qm);
			if (exiting) return;
			uiExecutingTask = execute_sync(*ef, MFF_NOWAIT);
		}
		while (!qsem_wait(finishSem, 50)) {
			if (exiting) {
				uiExecutingTask = -1;
				qsem_free(finishSem);
				return;
			}
		}
		uiExecutingTask = -1;
		qsem_free(finishSem);
		delete ef;

Admittedly polling the exiting flag is not elegant - but IDA does not offer a way to get event object signaling, and multiple object wait so something like this is sufficient at least to be safe.

Now you can just wrap any code into a lambda as an argument to std::function<void()> to bind it and very easily execute anything on the main thread taking into account data thread safety issues if any. This allows for cancellable long running operations.

Another area of concern is process creation using

	launch_process_params_t procInf;
...
	PROCESS_INFORMATION pi{};
	procInf.info = π
	void* p = launch_process(procInf, &errbuf);

IDA will leak thread handles on Windows unless you remember:

if (pi.hThread != INVALID_HANDLE_VALUE) CloseHandle(pi.hThread);

term_process and check_exit_process will both close the process handle for you.

Any uses of q functions with a create such as qthread_create must always have a matching free e.g. qthread_free and in this case should always have a qthread_join as well. Although qthread_kill is possible, it is a temporary workaround for deeper architectural issues or an emergency as the side effects nearly always outweigh the benefits and it could destabilize the IDA process.

Again all of these things were discovered, and not mentioned in any documentation.

[chants]

Here is the final clean code of the above which is polling-free since polling technically should not be done in a waiting loop ever in modern OS code where the OS can always schedule that much more safely and efficiently:

	/// Execute code in the main thread - to be used with execute_sync().
	struct execFunctor : public exec_request_t
	{
		std::function<void()> fun;
		qsemaphore_t finishSem;
		int* taskNo;
		execFunctor(std::function<void()> f, qsemaphore_t qs, int* tsk) : fun(f), finishSem(qs), taskNo(tsk) {}
		virtual ~execFunctor(void) override {}
		/// Callback to be executed.
		/// If this function raises an exception, execute_sync() never returns.
		virtual int idaapi execute(void)
		{
			fun();
			*taskNo = -1;
			qsem_post(finishSem);
			return 0;
		}
	};
	void IdaCallback::executeOnMainThread(std::function<void()> fun)
	{
		execFunctor* ef = new execFunctor(fun, di->termSem, &di->uiExecutingTask);
		{ //must be a locked unit otherwise race condition can occur
			qmutex_locker_t lock(di->qm);
			if (di->exiting) return;
			di->uiExecutingTask = execute_sync(*ef, MFF_NOWAIT);
		}
		qsem_wait(di->termSem, -1);
		if (di->uiExecutingTask == -1) delete ef;
	}

Forced termination (of course the qsem_create/qsem_free must be called at program initialization and termination):

		{ //must be a locked unit otherwise race condition can occur
			qmutex_locker_t lock(decompInfo->qm);
			decompInfo->exiting = true;			
			if (decompInfo->uiExecutingTask != -1) {
				if (cancel_exec_request(decompInfo->uiExecutingTask)) {
					qsem_post(decompInfo->termSem);
				}
			}
		}

Note that qwait_for_handles despite the docs not making it clear works for semaphores so you can do a multiple wait (WaitForMultipleObjects/select) e.g.:

int idx = -1;
qhandle_t handles[2] = { finishSem, termSem };
while (qwait_for_handles(&idx, handles, 2, 0, -1), idx == -1) {}

[chants]

We see in graph.hpp the following events:

  grcode_user_title,          ///< render node title of a user-defined graph.
                              ///< \param g               (::mutable_graph_t *)
                              ///< \param node            (int)
                              ///< \param title_rect      (::rect_t *)
                              ///< \param title_bg_color  (int)
                              ///< \param dc              (HDC)
                              ///< \retval 0  did not render, ida will fill it with title_bg_color
                              ///< \retval 1  rendered node title

  grcode_user_draw,           ///< render node of a user-defined graph.
                              ///< NB: draw only on the specified DC and nowhere else!
                              ///< \param g          (::mutable_graph_t *)
                              ///< \param node       (int)
                              ///< \param node_rect  (::rect_t *)
                              ///< \param dc         (HDC)
                              ///< \retval 0  not rendered
                              ///< \retval 1  rendered

But HDC is certainly NOT what you thought it might be - a windows handle to a device context as IDA is totally portable. So after taking a stack trace, decompiling IDA64.exe and tracing the argument, its in fact a QT::QPainter object pointer. Basically the QT equivalent of a device context.

https://doc.qt.io/qt-5/qpainter.html

This is another totally undocumented, even if logical feature present. Unfortunately you must compile with QT headers, should link with the exact QT libs that IDA uses so it will reuse the QT DLL included with IDA, in this case for QTPainter - Qt5Gui.dll version 5.6.3.0

[bongos_man]

Originally Posted by chants

Unfortunately you must compile with QT headers, should link with the exact QT libs that IDA uses so it will reuse the QT DLL included with IDA, in this case for QTPainter - Qt5Gui.dll version 5.6.3.0

this is not entirely true. you can in fact substitute ida's qt dlls with your own (potentially modified) dlls. the primary difference between the stock qt dlls and those bundled with ida is that hex-rays builds theirs with "-qtnamespace QT" and thus all symbols with non-C linkage are not mangled identically to stock qt. here is the original post (from 2011, time flies ;]) from when idaq.exe first shipped:

https://www.hexblog.com/?p=250

not much has changed since then, here is info for ida 7.3 (i can confirm that this works, including complicated cases such as plugins loaded by qt itself--of course these too must be built against an appropriately configured qt installation):

https://www.hexblog.com/?p=1341

if you want to build qt without so much useless garbage, this e.g. will work:

..\qt-everywhere-opensource-src-5.6.3\configure -prefix C:/Qt/5.6.3.0 -debug-and-release -force-debug-info -platform win32-msvc2017 -opensource -confirm-license -shared -no-compile-examples -nomake examples -nomake tests -nomake tools -opengl desktop -no-warnings-are-errors -ltcg -no-rtti -sse2 -sse3 -ssse3 -sse4.1 -sse4.2 -avx -no-ssl -no-openssl -no-dbus -no-audio-backend -no-wmf-backend -no-qml-debug -no-native-gestures -mp -skip qt3d -skip qtactiveqt -skip qtandroidextras -skip qtcanvas3d -skip qtconnectivity -skip qtdeclarative -skip qtdoc -skip qtenginio -skip qtgraphicaleffects -skip qtimageformats -skip qtlocation -skip qtmacextras -skip qtmultimedia -skip qtquickcontrols -skip qtquickcontrols2 -skip qtscript -skip qtsensors -skip qtserialbus -skip qtserialport -skip qtsvg -skip qttools -skip qttranslations -skip qtwayland -skip qtwebchannel -skip qtwebengine -skip qtwebsockets -skip qtwebview -skip qtx11extras -skip qtxmlpatterns -qtnamespace QT

ida 7.2 (can also confirm my own success):

https://www.hexblog.com/?p=1261

see hexblog for more entries for previous versions, but again not much changes from release to release regarding qt.