IDA SDK C++ tricks and tips

[chants]

Another interesting function is execute_sync with MFF_NOWAIT which is needed when a thread doing work may need to be interrupted to prevent deadlock.

THREAD_SAFE inline int execute_sync(exec_request_t &req, int reqf) { return callui(ui_execute_sync, &req, reqf).i; }

We see a exec_request_t object reference is passed, and then a pointer to this is passed along. The return value is the ID for cancelling:

THREAD_SAFE inline bool cancel_exec_request(int req_id)

Now if you want to cancel the request, such as in an interruption case, likely crashes will occur. This is because for whatever inexplicable reason, cancel_exec_request is calling delete on the exec_request_t pointer. So if you did not allocate it with new but just as a stack variable, it will be trying to free stack memory causing an obvious crash.

To deal with this, the best I could come up with was using both a semaphore to track the exec request completion, along with a termination flag and a mutex around that to prevent race conditions:
Global (hopefully stored somewhere referenceable from a singular global object):

		bool exiting = false;
		int uiExecutingTask = -1;
		qmutex_t qm = qmutex_create();

Main thread termination:

		{ //must be a locked unit otherwise race condition can occur
			qmutex_locker_t lock(qm);
			exiting = true;
			if (uiExecutingTask != -1) {
				cancel_exec_request(uiExecutingTask);
			}
		}

	struct execFunctor : public exec_request_t
	{
		std::function<void()> fun;
		qsemaphore_t finishSem;
		execFunctor(std::function<void()> f, qsemaphore_t qs) : fun(f), finishSem(qs) {}
		virtual ~execFunctor(void) override {}
		/// Callback to be executed.
		/// If this function raises an exception, execute_sync() never returns.
		virtual int idaapi execute(void)
		{
			fun();
			qsem_post(finishSem);
			return 0;
		}
	};
	void executeOnMainThread(std::function<void()> fun)
	{
		qsemaphore_t finishSem = qsem_create(nullptr, 0);
		execFunctor* ef = new execFunctor(fun, finishSem);
		{ //must be a locked unit otherwise race condition can occur
			qmutex_locker_t lock(qm);
			if (exiting) return;
			uiExecutingTask = execute_sync(*ef, MFF_NOWAIT);
		}
		while (!qsem_wait(finishSem, 50)) {
			if (exiting) {
				uiExecutingTask = -1;
				qsem_free(finishSem);
				return;
			}
		}
		uiExecutingTask = -1;
		qsem_free(finishSem);
		delete ef;

Admittedly polling the exiting flag is not elegant - but IDA does not offer a way to get event object signaling, and multiple object wait so something like this is sufficient at least to be safe.

Now you can just wrap any code into a lambda as an argument to std::function<void()> to bind it and very easily execute anything on the main thread taking into account data thread safety issues if any. This allows for cancellable long running operations.

Another area of concern is process creation using

	launch_process_params_t procInf;
...
	PROCESS_INFORMATION pi{};
	procInf.info = π
	void* p = launch_process(procInf, &errbuf);

IDA will leak thread handles on Windows unless you remember:

if (pi.hThread != INVALID_HANDLE_VALUE) CloseHandle(pi.hThread);

term_process and check_exit_process will both close the process handle for you.

Any uses of q functions with a create such as qthread_create must always have a matching free e.g. qthread_free and in this case should always have a qthread_join as well. Although qthread_kill is possible, it is a temporary workaround for deeper architectural issues or an emergency as the side effects nearly always outweigh the benefits and it could destabilize the IDA process.

Again all of these things were discovered, and not mentioned in any documentation.