I thought you've found the holy grail and it works
That was the reason why I've designed out the driver based protection from OllyExt. BTW I've made big steps to make my plugin multi-debugger capable. If you make a plugin framework for your debugger then I could make an X64DbgExt