swlepus, I suggest you to read the readme file on the PubKey_Replacer170_win folder, and study the flexlm sdk. I already wrote that second way is working... You need only to know how to work ida pro and how to build the new vendor with VS2008/2010/2012...
Here another part of code from a v11.4 sdk
Code:
if ((l_6counter == l_2086counter) && ((l_2082buff ^ 12052) & 0xff)) l_2082buff ^= 12052;
if ((l_6counter == (l_2086counter + 1)) && ((l_2082buff ^ 3205) & 0xff)) l_2082buff ^= 3205;
if ((l_6counter == (l_2086counter + 3)) && ((l_2082buff ^ 8108) & 0xff)) l_2082buff ^= 8108;
if ((l_6counter == (l_2086counter + 2)) && ((l_2082buff ^ 8083) & 0xff)) l_2082buff ^= 8083;
The values (not in hex) 12052, 3205, 8108 and 8083 are the 4 seeds for the handshake process.